tract

Why we included this project

tract is a Rust inference engine that takes ONNX, TFLite, or NNEF models, optimizes them, and runs them on whatever hardware you have: embedded ARM CPUs, NVIDIA and Apple GPUs, or WebAssembly in a browser. That flexibility comes from a translate-once design: you convert a model to tract's NNEF-based intermediate format at build time, then ship a tiny runtime that never loads protobuf or the training framework. Sonos has run it in production for wake-word and streaming speech-recognition, and the same engine also handles LLMs, text-to-image, and classical vision models, each with a runnable example. If low-latency, real-time inference is part of your workload, pulsification support is worth a look, since one model can serve both batch and live requests.