vllm

Why we included this project

vLLM is the serving engine most teams land on when they need to run large language models in production. It takes a Hugging Face model and exposes it as an OpenAI-compatible API, using continuous batching, prefix caching, and efficient key-value memory management to fit more concurrent requests on the same hardware. Model support spans a wide range of families and quantization formats, and it runs on NVIDIA, AMD, and TPU accelerators, so it scales from a single GPU box to a multi-node cluster. If cost per token and latency are what you care about, the speculative decoding and disaggregated prefill options are worth evaluating. For anyone building an application that needs dependable, fast serving rather than a notebook experiment, this is a solid default.