cortex

Why we included this project

Once your models are trained, the hard part is getting them to serve real traffic, and Cortex was built specifically for that. It turns an AWS EKS cluster into a serving platform where the same model can run as a realtime endpoint, an async worker, or a one-off distributed batch job, with autoscaling driven by request volume or queue length instead of manual capacity planning. The cluster management layer handles the infrastructure chores that normally eat a platform team's week, from CPU and GPU node autoscaling to spot instance fallback and private VPC networking with IAM integration. You can provision clusters with declarative config or a Terraform provider, and metrics and logs flow into Prometheus, Grafana, or CloudWatch. One honest caveat: the original authors no longer maintain it actively, so weigh that before betting a long-term stack on it, even though the design is sound and the docs are thorough.