maestro

Why we included this project

Maestro is Netflix's workflow orchestrator for data and ML pipelines, and the repo is a rare chance to see how a company that large actually schedules work at scale. Workflows are DAGs with dependencies, triggers, retry policies, and conditional branching, and the engine keeps latency within a strict SLO even when thousands of workflows kick off at the same moment. Data engineers and ML engineers who have outgrown Airflow-style single-leader schedulers will find the interesting details in how it spreads execution across many nodes, handles workflows with tens of thousands of jobs, and runs each step in a container under the calling user's identity. The project ships a runnable server with a REST API and Docker support, so you can stand it up locally and explore the workflow-as-a-service model, versioned definitions, and shared job templates rather than just reading the blog posts.