vaex

Why we included this project

Most dataframe tools choke once your data stops fitting in RAM, and that is where Vaex shines. It reads files via memory mapping, so you can compute statistics, build histograms and density plots, and filter billions of rows without ever loading the full dataset into memory. Its lazy expression system lets you define derived virtual columns that are evaluated on the fly, keeping interactive work in Jupyter snappy even as the data stays on disk. That makes it a convenient alternative to spinning up a cluster for early-stage exploration, feature vetting before training a model, or ad-hoc analytics on HDF5 and Arrow files. The built-in plotting and Jupyter widget support are a nice bonus when you want to quickly eyeball large datasets.