#41 · Primary category： Inference & Local Deploy
Mooncake
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
Project last updated：08/29/26
GitHub Stars
6.4K
Forks
1.1K
Contributors
310
License
Apache-2.0
Why we included this project
Running LLMs at scale means keeping latency stable as traffic swings, and Mooncake is built for that job. Kimi, Moonshot AI's assistant, runs on top of it, so the design has proven itself under real, high-volume traffic rather than just in benchmarks. Mooncake splits prefill and decode across a cluster and passes the key-value cache over RDMA, which keeps a busy GPU from stalling on memory bandwidth or slow object storage. If you're already on vLLM or SGLang, the transfer engine drops in as an upgrade instead of forcing a rewrite, and the FAST'25 paper plus the open-sourced components give you a concrete starting point for tuning on your own hardware. Expect some distributed-systems work up front; the payoff is more tokens per second from the GPU fleet you already own.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
ollama
Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
llama.cpp
LLM inference in C/C++
vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
gpt4all
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
LocalAI
LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.