petastorm

Why we included this project

Petastorm is a data access library for training deep learning models directly from Apache Parquet files, so you can keep your training data in columnar storage instead of converting it to framework-specific record formats. It works with TensorFlow, PyTorch, and PySpark, and also from plain Python, which makes it a convenient bridge between an existing data lake and model code. The reader handles the details that usually eat engineering time in real workloads: selective column readout, shuffling, row filtering, partitioning for multi-GPU runs, and local caching. If your team already stores features or training data as Parquet and wants a single loading path across frameworks, this removes a lot of glue code.