#42 · Primary category: AI Data Infrastructure & Storage
ffcv
FFCV: Fast Forward Computer Vision (and other ML workloads!)
Project last updated:06/16/24
GitHub Stars
3.0K
Forks
180
Contributors
31
License
Apache-2.0
Why we included this project
Dataloading is a classic bottleneck in vision training: the GPU sits idle waiting for images to be decoded off disk. FFCV sidesteps that by converting a dataset into a custom binary format and serving it through a highly optimized PyTorch loader, so the GPU stays fed. The speedup is dramatic in practice, with ImageNet-scale training finishing in roughly half an hour on a single GPU and benchmark costs dropping accordingly. Since you keep the same training loop and only swap in the loader, adopting it is fairly painless if you already work in PyTorch. If you run the same large vision datasets over and over, that saved iteration time adds up quickly.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
ClickHouse
ClickHouse® is a real-time analytics database management system
simdjson
Parsing gigabytes of JSON per second : used by Facebook/Meta Velox, the Node.js runtime, ClickHouse, WatermelonDB, Apache Doris, Milvus, StarRocks
gun
An open source cybersecurity protocol for syncing decentralized graph data.
emqx
The most scalable and reliable MQTT broker for AI, IoT, IIoT and connected vehicles
server
MariaDB server is a community developed fork of MySQL server. Started by core members of the original MySQL team, MariaDB actively works with outside developers to deliver the most featureful, stable, and sanely licensed open SQL server in the industry.