willow-inference-server

Why we included this project

This server keeps speech recognition and synthesis on your own hardware, so audio never has to leave the building for a cloud API. It is built around Whisper models for near-real-time transcription and bundles text-to-speech for assistant-style replies, and it is tuned to run on modest CUDA cards such as the GTX 1060 as well as beefier ones. The WebRTC path is the interesting part: you can stream audio from a browser or device and leave the session open for days while keeping bandwidth low, which makes it a good fit for always-on home or office setups. You also get direct control over which Whisper models load and how much VRAM they use, so if you want low-latency, private ASR on hardware you already own, this is worth a close look.