dagster

Why we included this project

Dagster is built around a simple idea: you declare the data assets you want, tables, datasets, models, reports, as plain Python functions, and it handles the rest. The platform works out dependencies between those assets, manages partitions and incremental runs, and keeps a live catalog of what exists and how each piece was produced. That model pays off when your work spans analytics tables, feature stores, and trained models, because lineage and freshness checks travel with the data instead of living in a separate scheduler. Teams already running dbt, Spark, Databricks, or Snowflake alongside ML workloads get observability and metadata tracking that make debugging and production monitoring less painful. It scales from a single pipeline to a shared platform with data quality checks and cost visibility, which makes it worth evaluating if you are picking an orchestrator.