argo-workflows

Why we included this project

Argo Workflows runs multi-step compute jobs on Kubernetes, with each step of a pipeline defined as a container. You can lay out a job as a plain sequence of tasks or as a directed acyclic graph, which makes dependencies and parallel branches easy to express without writing a custom scheduler; retries, timeouts, and conditionals are built in. Machine learning teams use it to chain data preparation, training, evaluation, and serving on the same cluster, and data engineers lean on it for batch and ETL workloads. As a CNCF graduated project it has a large ecosystem around it, including integrations with Kubeflow Pipelines, Katib, and Seldon, plus a UI, a REST/gRPC API, and a Python SDK for defining and monitoring runs. If your workloads are already containerized and you want an engine that scales with Kubernetes itself, this is a strong fit.