llm.c

Why we included this project

Most people who train transformers never see past the PyTorch call stack. llm.c strips that away: it implements GPT-2 and GPT-3 pretraining in plain C and CUDA, with a roughly 1,000-line fp32 CPU reference sitting next to the faster CUDA mainline. Because it is a real training loop you can build with make and step through in a debugger, it works well as a way to learn CUDA kernels, mixed precision, and multi-node training. Teams that want to reproduce a known model run without pulling in a heavyweight framework get a compact, auditable baseline, and the parallel PyTorch implementation makes it easy to check that the C version agrees with a familiar training script.