text-dedup

Why we included this project

Duplicate and near-duplicate text is a quiet problem in training data: it inflates corpus size, wastes compute, and can leak test examples into training. text-dedup bundles the standard deduplication algorithms (MinHash with LSH, SimHash, suffix-array substring matching, and Bloom filters) behind one TOML-driven interface, so you can pick the approach that fits your data without wiring up each library yourself. It reads local parquet files and Hugging Face datasets and writes back a cleaned version, which makes it a practical step in any large-corpus pipeline. The config files keep runs reproducible, and the included benchmarks on CORE and NEWS-COPY give you a concrete read on precision, recall, and runtime before you commit. Teams preparing corpora for LLM pretraining or fine-tuning will find it a solid, self-contained starting point.