LMCache

Why we included this project

LMCache tackles the cost of recomputing key-value caches for repeated prompt prefixes in LLM serving. It stores those caches across CPU memory, local disk, Redis, and object storage, then reuses them across requests, sessions, and separate engine processes. That lowers time-to-first-token and eases GPU memory pressure, and because it runs as a standalone daemon it works alongside vLLM and similar engines without forcing a rewrite. The payoff is most visible for teams serving the same documents to many users or running agentic loops with shared context. Its pluggable storage backends and a KV transformation interface also make it practical to test against existing infrastructure and to prototype compression or token-dropping ideas.