#11 · Primary category： Inference & Local Deploy
airllm
AirLLM 70B inference with single 4GB GPU
Project last updated：08/29/26
GitHub Stars
33.1K
Forks
3.5K
Contributors
10
License
Apache-2.0
Why we included this project
Running a 70B model usually means renting a big GPU, but AirLLM takes a different path: it cuts inference memory so hard that the same model fits on a single 4GB card, without quantizing, distilling, or pruning, so the weights stay untouched. The trick scales too. Sparse MoE checkpoints like DeepSeek-V3 and Kimi K3 stream one expert at a time instead of a whole layer, which is how a 2.8T-parameter model can run in under 4GB and Llama 3.1 405B fits on 8GB. For developers and small teams who want to try frontier-scale open models on hardware they already own, that removes a real barrier. Just know it is an inference library, not a full serving stack, so you will be wiring it into your own pipeline.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
ollama
Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
llama.cpp
LLM inference in C/C++
vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
gpt4all
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
LocalAI
LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.