#35 · Primary category： Inference & Local Deploy
PowerInfer
High-speed Large Language Model Serving for Local Deployment
Project last updated：05/11/26
GitHub Stars
9.8K
Forks
597
Contributors
399
License
MIT
Why we included this project
PowerInfer targets the awkward middle of local AI work: a machine with a single consumer GPU where a full server setup is overkill. It exploits activation locality, the observation that most token generation depends on a small set of consistently active 'hot' neurons. Those get preloaded onto the GPU, while the rarer 'cold' neurons are computed on the CPU, which keeps VRAM pressure and CPU-GPU data transfer low enough to serve something like Falcon-40B on one RTX 4090 at usable speeds. People coming from llama.cpp will recognize the interface, since most example workflows carry over and it loads llama.cpp model weights. That makes it a practical option for local serving and prototyping on limited hardware, and for poking at sparse ReLU-based models.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
ollama
Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
llama.cpp
LLM inference in C/C++
vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
gpt4all
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
LocalAI
LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.