data-juicer

Why we included this project

Data-Juicer is for anyone whose time goes to cleaning data instead of training models. It includes more than 200 composable operators for cleaning, filtering, deduplicating, and restructuring text, image, audio, and video datasets headed into foundation-model work. Pipelines are defined as versioned YAML recipes, so a whole processing run can be committed and shared like code, which makes reproducible data prep practical for a team. The same operators run on a laptop or across a Ray cluster, so a quick prototype and a multi-terabyte pre-training corpus use the same building blocks. People preparing data for pre-training, instruction tuning, RL, RAG, or even embodied-agent traces get working pieces rather than writing glue code.