volcano

Why we included this project

Kubernetes' default scheduler handles long-running services well, but batch and distributed training jobs often stall waiting for a full set of pods to be schedulable. Volcano replaces kube-scheduler's default behavior with a batch scheduler built for AI and HPC workloads: jobs start only when every pod they need can be placed, and fair-share queues, preemption, and oversubscription stop shared clusters from turning into a free-for-all. It integrates with the usual AI and data stack, including PyTorch, TensorFlow, Spark, Flink, Ray, MPI, and Kubeflow, and places work with topology in mind so GPU and MPI communication stay fast. Job controllers and a CLI let data scientists submit work without hand-writing YAML. If you run a mix of ML, big-data, or genomics workloads on a shared cluster, this CNCF project gives you one way to manage priorities and quotas across teams without tuning each scheduler by hand.