#144 · Primary category： Inference & Local Deploy
turboquant
TurboQuant: Near-optimal KV cache quantization for LLM inference (3-bit keys, 2-bit values) with Triton kernels + vLLM integration
Project last updated：03/27/26
GitHub Stars
1.7K
Forks
195
Contributors
1
License
GPL-3.0
Why we included this project
Serving teams that run out of GPU memory on long prompts will find a direct fix here: TurboQuant shrinks the KV cache by quantizing keys to 3 bits and values to 2 bits using Triton kernels, then hooks into vLLM so the change doesn't force a rewrite of your serving stack. On the benchmarks included in the repo, that roughly doubles the token capacity of the same GPUs while keeping prefill and decode throughput in the same range, across both dense and MoE models on RTX 3090 and 5090 hardware. The memory savings translate into more concurrent sequences or longer documents per request, which is the usual bottleneck for teams serving big contexts. The README also reports accuracy results honestly, including needle-in-a-haystack tests, so you can judge the trade-off before pointing production traffic at it.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
ollama
Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
llama.cpp
LLM inference in C/C++
vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
gpt4all
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
LocalAI
LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.