#136 · Primary category： Inference & Local Deploy
OnnxStream
Lightweight inference library for ONNX files, written in C++. It can run Stable Diffusion XL 1.0 on a RPI Zero 2 (or in 298MB of RAM) but also Mistral 7B on desktops and servers. ARM, x86, WASM, RISC-V supported. Accelerated by XNNPACK. Python, C# and JS(WASM) bindings available.
Project last updated：06/18/26
GitHub Stars
2.1K
Forks
99
Contributors
6
License
Other
Why we included this project
OnnxStream takes a different route than most ONNX runtimes: rather than loading a whole model into RAM, it streams attention weights from disk as they are needed. That design choice is what lets Stable Diffusion XL run on a Raspberry Pi Zero 2 or inside about 300 MB of memory, and it also lets Mistral 7B run on ordinary desktops and servers without a GPU. The C++ core targets ARM, x86, RISC-V, and WebAssembly, so the same codebase can span embedded boards and in-browser demos, with XNNPACK keeping CPU performance reasonable. You also get working examples for Stable Diffusion, Whisper, YOLOv8, and TinyLlama chat instead of a bare API, and Python, C#, and JavaScript bindings mean you do not have to write C++ to use it.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
ollama
Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
llama.cpp
LLM inference in C/C++
vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
gpt4all
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
LocalAI
LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.