kubeai

Why we included this project

If your team runs Kubernetes, KubeAI is a way to serve models without hand-assembling a stack. It's a Kubernetes operator: you declare a model in a CRD and it handles downloading weights, mounting storage, and scaling replicas from zero as demand arrives. The proxy is the part that stands out. Instead of kube-proxy's random load balancing, it uses prefix-aware routing that keeps vLLM's KV cache warm, which helps throughput and time-to-first-token when you run multiple replicas. It ships with a catalog of pre-configured models for common GPU types, runs on CPU, GPU, or TPU, and doesn't need Istio, Knative, or a Prometheus adapter. Teams already on Kubernetes who want to serve LLMs, embeddings, reranking, or speech-to-text will find this covers the whole path from model artifact to a working OpenAI-compatible endpoint.