serving

Why we included this project

When a trained model has to keep answering production requests without downtime, TensorFlow Serving is one of the more battle-tested options out there. It serves a saved model over both gRPC and HTTP, and it lets you roll out new versions, canary weights, and A/B test experimental models without changing any client code. It tracks multiple models and versions through a low-overhead, reference-counted lookup table, which keeps per-inference latency low and deployments predictable. Its scheduler can also batch individual requests for joint GPU execution, which helps when throughput matters more than per-request latency. If you already work in the TensorFlow ecosystem and want a serving layer that handles the messy lifecycle parts for you, this is worth studying before you build your own.