litgpt

Why we included this project

Teams that want to take an open-weight model beyond out-of-the-box prompting will find LitGPT unusually practical. It ships from-scratch, single-file implementations of 20+ architectures, covering Llama, Gemma, Qwen, Mistral, and Phi, with ready-made recipes for pretraining and finetuning using LoRA, QLoRA, and adapters. That combination matters when you need to train a model yourself instead of calling an API: the codebase is transparent, with no abstraction layers to fight through, and Flash Attention plus FSDP support let workloads stretch from a single consumer GPU up to thousand-plus node clusters. The same model code handles inference too, including quantization down to fp4/8, so you can validate a finetune and deploy it without switching tools. Anyone with PyTorch experience who has been put off by the plumbing of large-scale training should look here first.