kvcached

Why we included this project

GPU teams that run vLLM or SGLang often see their accelerators sit idle between traffic spikes. kvcached tackles that by giving the KV cache an OS-style virtual memory layer: an engine reserves virtual GPU memory up front and only maps physical memory when a request actually needs it. That separation lets several models share one GPU elastically, so allocations shift with bursty or mixed workloads rather than being pinned to each model's worst case. For deployments that want models to spin up and sleep on demand, this demand-driven approach directly raises utilization and trims serving cost. Since it installs as a plugin into existing engine environments, you can measure the effect on your own latency and throughput before committing.