ik_llama.cpp

Why we included this project

This fork of llama.cpp, started in mid-2024 and synced with upstream that August, focuses on faster CPU inference and ships additional quantization formats that in many cases beat the mainline build on speed. A number of inference features, including MLA, tensor parallel, multi-token prediction, and fused delta-net, appeared here before they reached llama.cpp proper. The project is upfront about its limits: CPU (AVX2 or newer, ARM_NEON) and Turing-or-newer CUDA are the maintained backends, while ROCm, Vulkan, and Metal get little attention, so plan around your hardware. You get the llama-server binary with OpenAI-compatible endpoints, Docker images, and support for recent Qwen, DeepSeek, GLM, and LLaMA models. Read the caveats about the -rtr repacking flag and Unsloth _XL models with f16 tensors before treating it as a drop-in replacement.