mars

Why we included this project

For teams that have outgrown single-machine pandas and scikit-learn, Mars offers a way to scale up without abandoning the code style they already know. Instead of forcing a rewrite around a new API, it exposes those familiar interfaces as lazy tensor, dataframe, and learn modules, so a computation that once crashed a laptop can run on a cluster with only modest code changes. The framework handles chunking, scheduling, and execution across cores or nodes, which makes it a practical fit for data preparation, feature engineering, and classical modeling like PCA or clustering on datasets that exceed memory. It also plugs into joblib and libraries such as XGBoost and LightGBM, so it slots into pipelines that already depend on the wider Python data ecosystem. If you want distributed data science without abandoning your current workflow, Mars is worth evaluating.