semantic-router

Why we included this project

Once you run more than one LLM provider, routing is where the quality, cost, and latency trade-offs actually land. This project turns that routing into a programmable layer: it reads request signals, user preferences, and application policies, then picks or composes the right model path per request, so you don't hard-code model choices into your application. Sitting in front of heterogeneous inference infrastructure, a single policy can keep private data on-prem while sending routine traffic to a cheaper cloud model, or route safety-sensitive prompts to a guard model. The OpenAI-compatible API and drop-in deployment modes, either as an Envoy ExtProc filter or a local service, mean existing clients keep working with minimal changes. If model selection across your stack needs to be repeatable and policy-driven, this is a concrete place to start.