dolma

Why we included this project

Training your own language model starts with raw web text that is noisy, repetitive, and full of duplicates, and Dolma is the toolkit that cleans it up. It processes billions of documents in parallel, with deduplication at document or paragraph level, quality and toxicity taggers (including Gopher and C4), and a mixing step that assembles the filtered output. It runs on a single laptop, a cluster, or S3-backed storage, so you can start small and scale as your data grows. The same repo also documents the 3-trillion-token Dolma corpus it produced for OLMo, so teams can inspect exactly how that dataset was built. Deduplication uses a Rust Bloom filter, which keeps things fast even when your input is measured in terabytes.