TopOpenSourceAI
zh

#59 · Primary category： Inference & Local Deploy

shimmy

api-server command-line-tool developer-tools gguf huggingface huggingface-models huggingface-transformers inference-server llama llamacpp llm-inference local-ai machine-learning ollama-api openai-compatible rust rust-crate transformers webgpu webgpu-shaders

⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.

Project last updated：08/29/26

View on GitHub

GitHub Stars

5.8K

Forks

561

Contributors

1

License

Apache-2.0

Why we included this project

Shimmy is a single Rust binary that serves GGUF models through OpenAI-compatible chat endpoints, so apps already speaking that API can point at it and skip the usual plumbing. Inference runs on a from-scratch WebGPU engine, which means it works on virtually any GPU, including integrated graphics, with no Python runtime or C++ toolchain in the picture. That combination fits privacy-minded users who want models running on their own hardware, and small teams doing local inference on laptops or constrained machines without a stack of dependencies. You download one binary, point it at a GGUF file, and existing OpenAI-style clients keep working as if nothing changed.

Articles for this project

No articles for this project yet.

To suggest a topic or contribute an article, contact us.

Related projects in this category

ollama

Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.

#1

llama.cpp

LLM inference in C/C++

#2

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

#3

gpt4all

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

#4

LocalAI

LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.

#5