#134 · Primary category： Inference & Local Deploy
paddler
Open-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale 🏓🦙 Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.
Project last updated：07/19/26
GitHub Stars
1.7K
Forks
97
Contributors
13
License
Apache-2.0
Why we included this project
Running your own LLM serving stack usually means juggling a lot of moving parts. Paddler trims that down to two: a balancer that distributes requests and agents that generate tokens and embeddings. It ships with the llama.cpp engine built in, so inference works without a separate backend. Request buffering lets agents scale up from zero hosts when traffic arrives, and agents can be added or removed on the fly, which pairs well with autoscaling tools. The whole thing is a single binary that runs on CPU or GPU, and it includes a web admin panel plus observability metrics for day to day operations. That makes it an appealing option for small DevOps teams that want predictable costs and full control over their data.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
ollama
Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
llama.cpp
LLM inference in C/C++
vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
gpt4all
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
LocalAI
LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.