#18 · Primary category： Inference & Local Deploy
ktransformers
A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations
Project last updated：08/28/26
GitHub Stars
19.3K
Forks
1.5K
Contributors
129
License
Apache-2.0
Why we included this project
KTransformers tackles a problem most people hit the moment they try to run a large Mixture-of-Experts model on their own hardware: the weights simply do not fit in GPU memory. Its answer is to split the work between CPU and GPU, keeping the experts that are actually being used on the accelerator and pushing the rest into system RAM, which lets models that normally demand several high-end GPUs run on a single consumer card. The project ships optimized CPU kernels with AMX/AVX support and INT4/INT8 quantization, plugs into SGLang for serving, and offers a LLaMA-Factory based fine-tuning path for MoE models on limited memory. It also tends to be among the first to support newly released open models like DeepSeek, Kimi, and GLM, with tutorials and benchmark numbers that show what the trade-offs actually look like.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
ollama
Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
llama.cpp
LLM inference in C/C++
vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
gpt4all
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
LocalAI
LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.