#160 · Primary category： Inference & Local Deploy
rwkv.cpp
INT4/INT5/INT8 and FP16 inference on CPU for RWKV language model
Project last updated：03/23/25
GitHub Stars
1.6K
Forks
129
Contributors
90
License
MIT
Why we included this project
This port of RWKV-LM onto the ggml runtime makes RWKV models practical on machines without a GPU. It ships as a C library with a convenient Python wrapper, and the core selling point is the range of quantized formats: INT4, INT5, INT8, and FP16, so you can shrink model files and speed up token generation on commodity CPUs by accepting a small quality loss. Teams running RWKV v4 through v7 on CPU-only or edge setups get a clear path from a Hugging Face checkpoint to a working local model, thanks to the included conversion and quantization scripts plus measured latency and perplexity tables for choosing a format. When a GPU is available, cuBLAS and hipBLAS support let you offload layers without switching runtimes.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
ollama
Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
llama.cpp
LLM inference in C/C++
vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
gpt4all
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
LocalAI
LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.