#152 · Primary category： Inference & Local Deploy
BigMoeOnEdge
Run MoE models bigger than your RAM. Frontier-size MoE on a 12 GB phone, CPU only, lossless, on stock llama.cpp
Project last updated：08/31/26
GitHub Stars
514
Forks
50
Contributors
2
License
Apache-2.0
Why we included this project
This project is for anyone who wants to run large mixture-of-experts models on hardware that doesn't have the RAM to hold them. It takes advantage of the fact that each token only activates a few of a model's experts, keeps the always-needed layers resident, and reads the requested experts from flash storage on the fly. The result is generation that's byte-identical to a fully in-RAM run, even when the model is several times larger than available memory. Since it's built on llama.cpp's public API rather than a fork, all existing quantizations, tokenizers, and chat templates work without modification, and picking up new upstream releases is just a submodule bump. If you're deploying DeepSeek-class or Qwen MoE models on phones or CPU-only PCs, this makes offline inference practical without a GPU.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
ollama
Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
llama.cpp
LLM inference in C/C++
vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
gpt4all
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
LocalAI
LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.