llama-cpp-python

Why we included this project

For running GGUF-format models entirely on your own hardware, this is the Python library most people pick to talk to llama.cpp. It exposes the C engine at two levels: a low-level ctypes interface for fine control, and a high-level API shaped like OpenAI's, so code written against one stays portable to the other. The bundled web server speaks the OpenAI protocol and adds function calling, vision support, and multi-model hosting on top. Because the package compiles llama.cpp itself, you choose the backend at install time, CPU, CUDA, Metal, Vulkan, or ROCm, which makes it workable from a laptop up to a dedicated GPU box. It also plugs into LangChain and LlamaIndex, so teams already using those can move to local inference without rewriting their orchestration code.