buun-llama-cpp

Why we included this project

Long context windows are usually the first thing to eat your VRAM when you run LLMs locally, and this llama.cpp fork attacks that problem directly. Its variable bit-rate KV cache starts at full FP16 precision and only compresses one layer at a time as memory pressure actually builds, so you get the best cache quality that fits at any given moment instead of paying a fixed 3-bit or 4-bit penalty up front. The codec ladder also includes trellis-coded quantization tiers that recover much of the quality normally lost at 2-3 bits. Because it's a drop-in llama.cpp server, you can point it at an existing GGUF model and try flags like `-ct vbr` without rewriting your stack. The maintainers call it highly experimental, so treat it as research-grade: good for evaluation and tuning, not a stable production default.