webdataset

Why we included this project

WebDataset exists to solve the I/O bottleneck that appears when your training data outgrows a plain folder. It packages samples into sharded tar archives and reads them as sequential streams, so loading stays fast whether the data is on a local drive or in a cloud bucket. Each tar entry groups everything for one sample under a shared basename, and numbered shards make it straightforward to split work across machines. The Python library slots into PyTorch's IterableDataset, letting you keep your normal decoding and augmentation steps while still handing the result to DataLoader as usual. Because media stays in its native file format, producing a WebDataset archive is as simple as creating a tar file. That combination of sequential I/O and simple packaging is why teams training large vision or language models reach for it instead of wrestling with thousands of loose files.