transformer-deploy

Why we included this project

Teams that run Hugging Face transformer models as part of a real service often hit the same wall: the checkpoint works in notebooks, but nobody knows what the actual latency will be. transformer-deploy shortens that gap by taking a model that can export to ONNX and, with one command, producing an optimized engine plus the full folder layout and Triton configuration, covering classification, token classification, dense embeddings, and text generation. The authors built it for a legal-domain semantic re-ranker that scores hundreds of snippets per query, so the benchmarks and defaults come from real latency pressure rather than toy examples. It handles the tedious wiring of TensorRT and ONNX Runtime, with quantization for CPU and GPU, so you get the speedup without becoming a compilation expert. If you are weighing Triton against a FastAPI-style server, the comparison tables and Docker-based demos let you decide on measured numbers instead of guesses.