Megatron-LM

Why we included this project

Megatron-LM is where much of the field's distributed training research has been validated, so if you are scaling a large transformer across many GPUs, this is a natural place to start. The repo holds two things: a reference implementation with pre-configured scripts for quick experiments, and Megatron Core, a composable library of GPU-optimized building blocks you can wire into your own training framework. The parallelism options are the main reason to look here. Tensor, pipeline, data, expert, and context parallelism, plus FP16, BF16, FP8, and FP4 mixed precision, let one codebase go from a few GPUs to a full cluster. Framework developers will find the transformer blocks and model architectures directly reusable, while research teams can lean on the ready-made examples to get a baseline run going fast. The companion Megatron Bridge converts checkpoints in both directions with Hugging Face, so moving between this stack and the wider ecosystem does not mean rebuilding weights.