kvpress

Why we included this project

The key-value cache in transformers grows linearly with input length, which makes long-context inference genuinely expensive: a single 1M-token prompt can eat hundreds of gigabytes of GPU memory. kvpress goes straight at that bottleneck with a set of prefill-time compression methods, called presses, that plug into existing Hugging Face models through a drop-in transformers pipeline, so you can try them without restructuring your serving setup. Each press is training-free and exposes a compression-ratio attribute, and the project ships a leaderboard and benchmarks comparing methods, so you can gauge the accuracy and latency trade-off before committing. That makes it a practical place to start, whether you want to apply an established method or prototype your own.