#156 · Primary category： Inference & Local Deploy
DeepSpeed-MII
MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.
Project last updated：06/30/25
GitHub Stars
2.1K
Forks
192
Contributors
36
License
Apache-2.0
Why we included this project
DeepSpeed-MII is the inference side of the DeepSpeed project: a Python library that serves large models on your own GPUs instead of sending them to a hosted API. It hides DeepSpeed's inference engine behind a simple interface, so HuggingFace models get continuous batching, blocked KV-caching, and tensor parallelism without you wiring those up yourself. For teams that self-host and care about tokens per second or strict latency, that can mean getting more work out of hardware they already own, and dense and Mixture-of-Experts transformers are both handled well. Since it sits on the same stack used in DeepSpeed training, teams already invested in DeepSpeed can reuse that optimization work instead of adopting a separate serving system.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
ollama
Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
llama.cpp
LLM inference in C/C++
vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
gpt4all
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
LocalAI
LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.