fastT5

Why we included this project

T5 models handle a lot of NLP work, but their sequential text generation gets slow, and the bigger the model the worse it gets. fastT5 goes after that directly: it converts a pretrained Hugging Face T5 model to ONNX, quantizes it, and runs it through onnxruntime, all from a single function call. The returned model still supports the familiar generate() method, so you keep your existing code. The payoff is a model about 3x smaller and inference up to roughly 5x faster on CPU, with beam search also seeing solid gains. The pipeline is customizable too, so you can run the export, quantization, and runtime steps separately if you want more control. It's a focused tool rather than a full serving platform, which makes sense for teams that want a quick, measurable speedup on modest hardware without retraining.