mllm

Why we included this project

mllm is a C++ inference runtime for running multimodal LLMs on phones and edge devices rather than in the cloud. It covers Arm CPUs, x86, Apple Silicon, and Qualcomm NPUs through QNN, with experimental CUDA support on Jetson boards. The engine pairs with a Pythonic, Torch-like API with eager execution and graph tracing, so you can define or adapt a model and compile it into a static representation that targets NPUs well. Around that core sit a C SDK, a Go-based CLI, and model converters for pulling weights from common formats, giving you the pieces needed to embed the runtime in an actual product. Documentation walks through build steps for Android, x86, macOS, and Docker, which is a useful way to gauge how far on-device inference goes on the hardware you own.