nano-vllm

Why we included this project

Most LLM serving engines are too large to read as a way of learning how they work. Nano-vLLM rebuilds the core of vLLM in roughly 1,200 lines of readable Python, so you can follow how prefix caching, tensor parallelism, torch compilation, and CUDA graphs fit together without digging through a huge production codebase. The API mirrors vLLM's LLM and SamplingParams interface, so an existing offline batch-inference script can be adapted with minimal changes. Its own benchmarks on a single consumer GPU show throughput on par with the full vLLM engine, which makes it more than a teaching project. Developers can study it as a reference before tuning their own serving stack, and teams can use it as a compact, auditable inference path for batch workloads.