vllm-turboquant

Why we included this project

Long-context vLLM serving that keeps running into memory limits is the use case here. The fork adds TurboQuant KV-cache quantization, which compresses the attention key and value cache so longer sequences or bigger batches fit on the same GPUs. The numbers are concrete: presets like FP8 keys with 4-bit values deliver roughly 2.6x cache compression at a modest perplexity cost, and the quantizer configuration is exposed so you can adjust bit widths and norm correction yourself. Because it tracks upstream vLLM closely, you keep the standard serving stack (continuous batching, PagedAttention, an OpenAI-compatible API) with the quantization layer on top. If you already run vLLM and want more context or throughput from existing hardware before buying more, this is a focused place to test that.