fastdup

Why we included this project

Getting a clean training set is usually the slowest part of any vision project, and fastdup attacks that bottleneck directly. Point it at a folder of images or video frames and it flags duplicates and near-duplicates, outliers, corrupted files, and clusters of visually similar shots, so you can prune a dataset before it ever reaches a training run. It does this on a single machine for tens of millions of images by working from compact similarity embeddings rather than raw pixels, and its gallery-style reports let you eyeball flagged groups and catch label errors without writing your own inspection scripts. For computer vision engineers and small data teams, it is a fast, scriptable first pass over raw visual data that would otherwise eat hours of manual review.