vortex

Why we included this project

Vortex is a columnar storage format built to sit under data processing engines, and it's worth a look from any team whose analytics or ML work is bottlenecked by scan speed. The project claims substantially faster reads and writes than Apache Parquet at similar compression ratios, and its zero-copy Arrow compatibility means it plugs into existing Arrow, DataFusion, DuckDB, Spark, Pandas, or Polars pipelines without rebuilding the data layer. The design separates logical schema from physical layout, with pluggable encodings, compression schemes, and layout strategies, so you can tune storage to your access patterns instead of living with Parquet's defaults. The file format itself is already declared stable, with backwards compatibility promised from release 0.36 onward, which makes early experimentation less risky. Teams keeping wide tables on object storage, where I/O tends to dominate compute, will probably want to run their own benchmarks before committing.