llm-d

Why we included this project

When LLM inference outgrows a single GPU and you're already running vLLM or SGLang on Kubernetes, llm-d acts as the orchestration layer above those model servers. It adds load-aware routing with prefix caching, offloads KV cache to CPU or disk so multi-turn sessions keep larger working sets, and supports prefill/decode disaggregation and wide expert parallelism for models like DeepSeek-R1. The project documents real benchmarks showing higher throughput, lower time-to-first-token, and better cluster utilization across NVIDIA, AMD, and Intel accelerators. It's a CNCF sandbox project founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA, which suggests the guides and Helm charts are built with production environments in mind. Teams beyond a single GPU looking for SLO-aware autoscaling and multi-tenant flow control can follow the well-lit path guides instead of tuning everything from scratch.