mosec

Why we included this project

Mosec takes the inference code you already have and turns it into an online API without asking you to relearn serving. It's written in Rust for the request layer and Python for the user-facing parts, so you keep working in the same language you use for training. The dynamic batching matters most: concurrent requests from multiple clients get grouped into a single batched call, which helps a lot when you're paying for GPU time. You can also split CPU, GPU, and I/O-heavy work into separate pipelined processes, so a slow pre- or post-processing step won't stall the whole service. It handles the serving concern and leaves model optimization and business logic to you, and it ships with Prometheus metrics and graceful shutdown for Kubernetes deployments.