vLLM-Moet

Why we included this project

Most frontier MoE checkpoints ship in a form that will not fit on consumer Blackwell cards, and this project attacks that problem directly. It pairs a patched vLLM serving stack with hand-written SM120 kernels that compress routed experts down to 2 bits, then adds an FP4 delta cache to recover precision, which is how GLM-5.2 (753B), DeepSeek-V4-Flash (159B), and Kimi-K2.7-Code (1T) end up servable on RTX 5090s and RTX PRO 6000 cards. The README backs that up with real numbers: measured decode and prefill throughput, needle-validated context windows, and a tiered expert residency scheme that spills to pinned host RAM and NVMe packs when VRAM runs out. It also builds on the official vLLM v0.24.0 release with a generated runtime patch rather than a fork, and ships prebuilt cubins for TP2/TP4, so teams already on vLLM can adopt it without maintaining a separate codebase. If you are trying to run frontier-scale MoE locally on Blackwell, this is one of the more concrete references around.