LitServe

Why we included this project

Most serving frameworks expect you to work inside their request pipeline, but LitServe takes the opposite approach: you define how inference actually happens in plain Python. A LitAPI subclass loads your models and spells out how each request becomes a prediction, while the framework handles the parts nobody wants to wire by hand, including request routing, batching, streaming, GPU placement, and scaling across workers. That freedom matters when you serve something more complex than a single stock LLM, such as multi-model pipelines, RAG endpoints, agents that call tools, or custom logic layered on a vLLM backend. Because it is built on FastAPI, teams already comfortable with that ecosystem can adopt it gradually, and the same server runs locally or moves to managed infrastructure as load grows. Engineers who want deployment control without writing MLOps glue code will find it a practical fit.