DataFlow

Why we included this project

DataFlow is an operator-based pipeline system for turning raw text, PDFs, and code into clean training data for LLM fine-tuning, pre-training, or RAG. Instead of writing one-off scripts, you compose cleaning and synthesis steps into reproducible pipelines, with ready-made flows that cover everything from text and math to code generation, plus large-scale PDF-to-QA conversion. Built-in data agents can recombine existing operators or draft new ones on demand, so a new data prep flow can be stood up without hand-coding everything. A visual WebUI and a Gradio interface let you build pipelines graphically rather than editing Python, and because the pipelines live in the Git/Python ecosystem, they are easy to version, share, and reuse across a small team. Data teams and ML engineers working with document-heavy datasets will find the emphasis on domain-specific high-quality data especially relevant.