llm-compressor

Why we included this project

Serving open-weight models with vLLM usually ends at the GPU-memory wall, and this library is built to push past it. You hand it a Hugging Face model, pick a quantization method from the bundled set (GPTQ, AWQ, SmoothQuant, AutoRound, plus the newer rotation-based approaches), and calibrate against a slice of your own data. Compression covers weight, activation, KV cache, and attention, and the result comes back in the compressed-tensors format vLLM loads directly, so what you get is a deployable checkpoint rather than a research artifact. For teams trying to trim cloud costs or fit a bigger model on the GPUs they already own, this is the shortest path from a full-precision checkpoint to a leaner one without hand-writing calibration code.