train-llm-from-scratch

Why we included this project

This is one of those repositories that actually shows you what happens inside an LLM rather than leaving the model as a black box. Everything is hand-written in plain PyTorch, from tokenizing raw text through building a transformer from the Attention Is All You Need paper, all the way to post-training with SFT, reward modeling, PPO, DPO, and GRPO, with no transformers, TRL, or PEFT libraries hiding the details. For engineers and students who want to see each mechanism as real code instead of a wrapper, that transparency is the main draw. The same scripts scale from a few million parameters to billion-parameter training on a single GPU, so you can run the pipeline yourself and watch how changing the data and the loss reshapes what the model produces. If you plan to fine-tune or align models later, working through this end-to-end journey builds intuition that documentation alone rarely gives you.