synthetic-data-kit

Why we included this project

Fine-tuning a model usually means your data has to arrive as user/assistant turns, and most source documents look nothing like that. This CLI bridges the gap: it ingests PDFs, HTML, Word files, slides, YouTube transcripts, and plain text, then calls an LLM backend you already run (a local vLLM server, for instance, or any API endpoint) to generate QA pairs or chain-of-thought reasoning examples. A curation step uses Llama itself as a judge to score the output and filter out weak examples before they reach your final set. The last command exports to common fine-tuning formats like Alpaca or OpenAI's, so the results drop straight into your training pipeline. If you already have an LLM endpoint and just need a dependable way to turn raw documents into labeled training data, this covers the whole trip without a heavyweight platform.