mup

Why we included this project

Hyperparameters that work on one model size often stop working when you scale up, and each growth step can turn into another round of tuning. This package from Microsoft Research implements maximal update parametrization, a way of setting initialization and learning-rate scalings so that optimal settings stay stable as a model gets wider. The payoff for people training large transformers or other very wide networks is that you can tune a small base model and carry those hyperparameters over to the full-size version with far less guesswork. The library slots into PyTorch models through a few targeted changes: an output readout module, mup's own init functions and optimizers, and a one-line call to attach base shapes. It also ships coord-check tooling to confirm the parametrization is applied correctly, which matters because silent mistakes here produce misleading results.