lmdeploy

Why we included this project

LMDeploy bundles what you need to move an open model from training into production serving: TurboMind, a fast inference engine with CUDA kernels tuned for throughput, quantization tooling covering 4-bit compression and FP8 for MoE models, and an OpenAI-compatible API server so existing clients keep working without rewrites. It runs InternLM, DeepSeek, Qwen, and Llama-family models on NVIDIA GPUs, with docs covering offline batch inference, CLI chat, and REST serving paths. Its published benchmarks also make it a useful yardstick when comparing serving frameworks, since it turns in strong numbers against alternatives like vLLM while shipping its own quantization workflow.