kserve

Why we included this project

Running models in production on Kubernetes usually means assembling your own serving layer. KServe handles that layer for you, with a single InferenceService abstraction that covers both generative LLMs and classic predictive models, so a vLLM-backed model and an XGBoost predictor share one control plane instead of separate stacks. Built on Kubernetes primitives and Knative, it brings request-based autoscaling, scale-to-zero, canary rollouts, and inference graph routing without extra machinery, and it adds model explainability and drift monitoring that bare model servers rarely provide. As a CNCF incubating project with an OpenAI-compatible protocol, it is a solid self-hosted default that does not tie you to one runtime.