rotorquant

Why we included this project

Long-context inference on a single GPU tends to exhaust KV cache memory before compute, because that cache scales with batch size and sequence length. RotorQuant attacks this directly by quantizing the key/value cache with small block-diagonal rotations that decorrelate vectors before scalar quantization and undo the rotation on decode, yielding about 5x to 10x compression with a small perplexity cost. Because the rotation blocks are small and independent, it skips the expensive butterfly transform that similar methods rely on, which is why decode and prefill speed up. For anyone already using llama.cpp, adoption is simple: point a patched build at cache-type flags like iso3 or planar3, and you get longer context or larger batches on the same hardware. This is a focused optimization library.