mixtral-offloading

Why we included this project

Mixtral-8x7B is a 46.7-billion-parameter model, too big for the memory on most home computers. This project makes it practical by pairing HQQ mixed quantization, which applies different precision to attention layers and experts, with per-expert MoE offloading, where each expert returns to the GPU only when a token needs it and an LRU cache keeps recently used experts in memory to cut transfers when processing adjacent tokens. The combination fits Mixtral on a single consumer GPU or a free Colab session. If you want to experiment with a capable open mixture-of-experts model without a multi-GPU workstation, the demo notebook is a good place to start, though there is no command-line tool yet and you will have to adapt the notebook for your own use. The arXiv technical report lays out the methods and measured results so you can weigh the trade-offs against your own hardware, with the caveat that a few techniques it describes are still missing from the repo.