Curator

Why we included this project

Preparing data for LLM pretraining or fine-tuning usually means writing the same cleanup scripts again and again, from scrubbing noisy web crawls to deduplicating documents. NeMo Curator packages that work as repeatable pipelines that run equally well on a laptop and on a multi-node Ray cluster, so you can test locally and scale up without rewriting anything. The GPU acceleration is the practical payoff, turning jobs that would crawl on a single machine into distributed runs. It also covers image, video, and audio curation, which keeps multimodal and speech work in the same codebase instead of forcing a separate toolchain.