#115 · Primary category： Inference & Local Deploy
distributed-llama
Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.
Project last updated：07/05/26
GitHub Stars
3.0K
Forks
248
Contributors
15
License
MIT
Why we included this project
Distributed Llama strings together spare home machines, including Raspberry Pis, into a single inference cluster. It splits model weights across nodes using tensor parallelism and syncs over Ethernet, so adding devices actually speeds up generation rather than just scaling capacity. That means you can run checkpoints in the 70B or 405B range on hardware no single box could load into memory, without paying for cloud GPUs. Setup comes down to one command that downloads the model and tokenizer, and the README covers Linux, macOS, Windows, and GPU runs, with real benchmarks posted in the discussions. Anyone self-hosting LLMs on consumer hardware, or curious how tensor parallelism behaves beyond one GPU, will find a working example worth studying.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
ollama
Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
llama.cpp
LLM inference in C/C++
vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
gpt4all
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
LocalAI
LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.