img2dataset

Why we included this project

Turning a list of image URLs into a usable training set is fiddly work, and img2dataset exists to automate the dull parts. It downloads images at high concurrency, resizes them, and writes the results out as webdataset tar shards or local folders, with captions saved when your source includes them. The included runnable examples are a real strength: they show concrete commands and time estimates for datasets from mscoco's 600k pairs up to LAION-5B's billions, so you can sanity-check your hardware before committing. It also honors X-Robots-Tag opt-out headers by default, which matters if you're scraping at scale and want to stay on the right side of site operators. One machine can handle roughly 100 million URLs in about 20 hours, which is the kind of number that tells you the tool was built for production use.