omlx

Why we included this project

oMLX is a self-contained LLM inference server built for Apple Silicon, and it is one of the few that feels finished rather than assembled. A single process serves text models, vision-language models, embeddings, and rerankers behind an OpenAI-compatible endpoint, so tools like Claude Code or Cursor connect with minimal setup. The tiered KV cache is the real differentiator: hot blocks stay in RAM while cold ones move to SSD, and the cache survives context changes and server restarts, so long coding sessions do not recompute from scratch. Continuous batching keeps throughput steady under concurrent requests, and the menu-bar app plus admin dashboard let you load, pin, and evict models visually instead of editing config files. For anyone running local models on a Mac who wants caching depth without assembling a stack of separate tools, this is worth a look.