#28 · Primary category： Inference & Local Deploy
server
The Triton Inference Server provides an optimized cloud and edge inferencing solution.
Project last updated：08/28/26
GitHub Stars
10.9K
Forks
1.8K
Contributors
149
License
BSD-3-Clause
Why we included this project
Most training frameworks stop at the artifact, so production teams often end up writing their own serving glue. Triton is the layer that takes a trained model and actually answers requests at scale, letting you host a mix of frameworks, from TensorRT and PyTorch to ONNX and Python backends, behind one API instead of standing up a separate runtime for each. Dynamic batching, concurrent model execution, and ensemble pipelines push more throughput out of your GPUs and CPU nodes without hand-rolling scheduling logic. It serves over HTTP/REST and gRPC, and a C and Java API lets you embed it into an edge application rather than run it as a standalone service. If your serving needs span cloud, data center, and edge hardware, this is a proven foundation with years of active development behind it.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
ollama
Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
llama.cpp
LLM inference in C/C++
vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
gpt4all
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
LocalAI
LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.