Adan

Why we included this project

Adan is a drop-in optimizer for PyTorch: you swap it in for Adam or AdamW with a couple of lines of code, keep your weight-decay and gradient-clipping settings, and get faster convergence without redesigning the training loop. It implements the Adaptive Nesterov Momentum algorithm from the paper published in IEEE TPAMI, and the repo includes both a standard version and a fused CUDA variant that keeps memory use down when training large models. The results span a wide range of architectures, from vision transformers and ResNets to BERT, GPT-2, and MoE training, and several downstream projects, including text-to-3D work like DreamFusion and Consistent3D, already use Adan as their default optimizer. If you are willing to raise the learning rate a bit, the documented recipes and reproducibility notes make it straightforward to test whether the speedup carries over to your own model.