mixture-of-experts

Why we included this project

For PyTorch developers who want to scale up a transformer's parameter count without a matching jump in compute, this library provides a concrete sparsely-gated Mixture of Experts layer. It handles top-2 gating, capacity balancing, and the auxiliary loss that keeps routing stable during training, and it includes a hierarchical two-level variant for GShard-style setups. You can also slot in your own expert networks rather than using the built-in defaults. One thing to know: the author now points people to the newer st-moe-pytorch for production work, so this project mostly serves as a compact, readable reference for how the sparsely-gated mechanism works in code.