airflow

Why we included this project

If your team runs data or machine-learning pipelines that need to fire on a schedule, retry on failure, and leave an audit trail, Airflow is the tool most teams reach for first. You define workflows as Python code, where each task is a step in a directed acyclic graph, and the platform handles the scheduler, the worker queue, and a web UI for inspecting task status and logs. It ships with a large library of operators that talk to cloud services, databases, and data warehouses, so you can wire together extraction, transformation, and model-training steps without hand-rolling glue code. Because pipelines are just Python, you can generate them dynamically and version them like any other codebase. Teams that already run Airflow in production will find a mature ecosystem of plugins, integrations, and community knowledge to lean on.