#56 · Primary category： Inference & Local Deploy
cactus
Quantization, kernels, runtime and inference engine for mobiles, wearables, smart home and robots.
Project last updated：08/26/26
GitHub Stars
6.0K
Forks
497
Contributors
52
License
Other
Why we included this project
Cactus is a full on-device inference toolchain rather than a single serving binary, aimed at teams running LLMs on phones, wearables, smart home gear, and robots. It pairs a rotation-based quantization scheme with CPU/GPU kernels tuned for Apple, Samsung, and Pixel silicon, then exposes text, speech, and vision through an OpenAI-compatible API. A transpiler turns custom PyTorch models into the engine's zero-copy computation graph, low-confidence local requests can hand off to a cloud model when device memory is tight, and built-in RAG grounds answers in local text files. The bundled CLI covers model conversion, download, and benchmarking on real hardware. For mobile developers who want offline, low-latency inference without wiring together separate quantization, kernel, and serving libraries, this is a practical shortcut.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
ollama
Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
llama.cpp
LLM inference in C/C++
vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
gpt4all
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
LocalAI
LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.