sglang-omni

Why we included this project

Serving TTS and ASR models behind an API is a different problem from hosting chat models, and this framework is designed around that difference. It splits generation into coordinated stages, gives each stage its own scheduler, and moves tensor data between workers over shared memory and cluster transports, which is what keeps streaming speech responsive when load rises. You get OpenAI-compatible endpoints for chat, /v1/audio/speech, batch and streaming generation, uploaded voices, and /v1/audio/transcriptions, with Qwen3-TTS, Qwen3-ASR, Higgs Audio, Fish Speech, and transcription with diarization supported directly. If you run a speech product, a voice assistant, or a multimodal service and would rather maintain one serving layer than wire several engines together, this is a practical option to try. Builders of their own speech serving stacks can also study the multi-stage design as a working example of scheduling and transport choices.