fastllm

Why we included this project

fastllm's headline feature is running a full 671B DeepSeek on a single GPU with more than 10GB of VRAM, offloading MoE layers to CPU. It is a C++ inference engine with its own operators, so it does not depend on PyTorch at runtime, and it serves dense models like Qwen and Llama as well as MoE models like DeepSeek. Tensor parallelism works across multiple cards, including odd counts like three or five, and hardware support is broad, covering older NVIDIA cards, AMD GPUs through ROCm, and several domestic accelerators, with FP8 inference available even on older devices. The pip-installable CLI offers a chat mode, a WebUI, and an OpenAI-style server, plus a terminal deployment wizard. On a dual-socket 9004/9005 server with a single GPU, it runs the full-precision DeepSeek at about 20 tokens per second, the INT4 version at 30, and past 60 when handling multiple concurrent requests.