MoBA

Why we included this project

Full attention grows quadratically with sequence length, which is why training on very long contexts gets painfully expensive. MoBA borrows the gating idea from mixture-of-experts and applies it to attention: each query token learns which blocks of the key-value cache actually matter, rather than being locked into fixed window or sink structures. That makes it worth a look for researchers and training engineers working on long-context models, and the repo is a transformers-friendly attention module with two backends, a naive one for visualizing block selection and an optimized one reporting a 40x speedup, plus unit tests and an example Llama script. The honest caveat is that MoBA is not a drop-in sparsity trick: the authors state that you must continue training a model to see the acceleration, so pretrained checkpoints can't just swap it in. It has already shipped in production for Kimi's long-context requests, a useful signal that the approach holds up beyond the paper.