#45 · Primary category： Inference & Local Deploy
mistral.rs
Fast, flexible LLM inference
Project last updated：08/29/26
GitHub Stars
7.6K
Forks
689
Contributors
96
License
MIT
Why we included this project
mistral.rs is a Rust-native inference engine for running LLMs on your own hardware rather than paying per token to a hosted API. A single binary covers in-process inference from Python or Rust, terminal chat, and OpenAI- and Anthropic-compatible HTTP endpoints, so you can prototype in a local session and later serve the same model as a drop-in replacement for a vendor API without changing client code. The engine includes paged attention, prefix caching, quantization to cut memory use, and an agentic runtime with tool hooks and local Python code execution. Teams building local-first apps or testing different serving setups will find it practical, and it is under active development.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
ollama
Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
llama.cpp
LLM inference in C/C++
vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
gpt4all
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
LocalAI
LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.