llm-foundry

Why we included this project

Teams building large language models need more than a model architecture; they need a pipeline that turns raw text into something a GPU cluster can chew through efficiently. llm-foundry covers that span, from converting source data into MosaicML's StreamingDataset format through pretraining models between roughly 125M and 70B parameters, finetuning on custom data, running academic or custom evaluations, and exporting the result for inference. It builds on the Composer trainer, and the configuration files in the repo reflect work that actually ran: Databricks' Mosaic team trained DBRX with these exact tools, so the YAML configs are patterns proven on frontier-scale workloads rather than toy examples. You also get benchmarking utilities for throughput and MFU plus Flash Attention support out of the box, which makes it a solid starting point for teams that want reproducible, GPU-efficient training without assembling the pieces themselves.