sonar

Why we included this project

Sonar is an inference engine for Hugging Face-compatible text and multimodal models, built as a fork of vLLM and aimed at teams that want to run high-traffic serving on their own hardware. It bundles the machinery that makes that practical: continuous batching, paged KV-cache management, prefix caching, speculative decoding, and quantization down to an FP8 KV cache, all exposed through an OpenAI-compatible API. It can also span multiple nodes without needing a Ray cluster, and the installers cover AMD ROCm, CPU, and Apple silicon alongside the usual CUDA path. The project already carries production traffic for the Dolphin Inference Network and PygmalionAI, which is a good sign that the harder corners have been worked out. For self-hosted serving with broad hardware and model coverage, it's worth a closer look.