torchscale

Why we included this project

TorchScale is a PyTorch library from Microsoft where several well-known model architecture innovations live as working code, including RetNet, LongNet, BitNet, and X-MoE. Instead of reimplementing attention or normalization yourself, you can switch these designs in through config flags, which makes it a practical starting point for reproducing papers or comparing alternative LLM backbones before committing to one. The code is reasonably concise and installs with a single pip command. It is primarily a collection of architecture primitives and research code rather than a turnkey pretrained model you can deploy directly, so plan to build on it rather than use it out of the box.