mlc-llm

Why we included this project

MLC LLM is a compiler-based inference engine that turns large language models into optimized native code for the hardware you actually have, from NVIDIA and AMD GPUs to Apple silicon, Intel GPUs, phones, and browsers. That makes it a strong pick for teams that want to run models on-premises or on edge devices rather than renting cloud GPUs, since the same model can be compiled once and deployed across very different targets. It also exposes an OpenAI-compatible API through a REST server, with bindings for Python, JavaScript, iOS, and Android, so it drops into existing tooling without much friction. If you care about getting good performance out of consumer or mobile hardware, or keeping inference fully self-hosted, this is worth a close look.