bytewax

Why we included this project

Bytewax brings stateful stream processing to Python, so you don't have to rebuild your data work in Java or Scala. You write pipelines as dataflow graphs in ordinary Python, and a Rust-based runtime handles distribution and automatic state recovery. Teams use it for online machine learning, real-time feature engineering, and continuous ETL where events arrive from Kafka, filesystems, or WebSockets, which lets models and features update as events land instead of in a nightly batch. It also fits the wider ML ecosystem: pair it with River for streaming anomaly detection, or use it to drive the embedding and chunking steps in a live RAG pipeline. One caveat worth knowing: the project is now community-maintained, so factor that in before committing to a long-lived production deployment.