#147 · Primary category： Inference & Local Deploy
ome
Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton
Project last updated：09/01/26
GitHub Stars
502
Forks
94
Contributors
39
License
Apache-2.0
Why we included this project
OME is a Kubernetes operator that takes over the operational side of LLM serving, so teams don't end up maintaining a pile of hand-rolled deployment scripts. Models are first-class custom resources: the operator parses architecture, parameter count, and capabilities straight from the model files, then scores the available runtimes (SGLang, vLLM, TensorRT-LLM, Triton) to pick a good match. It also handles the parts that usually bite in production, including GPU bin-packing with dynamic re-optimization, prefill-decode disaggregation, multi-node inference, and canary or blue-green rollouts. Integrations with Kueue, KEDA, and the Gateway API cover gang scheduling, autoscaling, and traffic routing, and a BenchmarkJob resource handles load testing. If you run a GPU cluster and want a control plane that manages models from download through serving to performance evaluation, OME is worth a look.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
ollama
Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
llama.cpp
LLM inference in C/C++
vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
gpt4all
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
LocalAI
LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.