MeZO

Why we included this project

Most fine-tuning runs hit the same wall: backpropagation's memory cost. MeZO sidesteps it by estimating gradients from forward passes alone, so you can train a 30-billion-parameter OPT model on a single A100 where Adam would top out around 2.7B. The results land within a few percent of standard fine-tuning on many tasks while using up to 12x less memory, and it works with LoRA and prefix tuning if you want to go further. Because it doesn't rely on differentiability, you can even optimize objectives like accuracy or F1 directly. The repo includes separate harnesses for RoBERTa and OPT experiments from the NeurIPS 2023 paper, plus guidance for wiring MeZO into existing HuggingFace code.