MEGABYTE-pytorch

Why we included this project

MEGABYTE-pytorch is a compact, readable implementation of the MEGABYTE architecture, a multiscale transformer that processes sequences up to a million bytes without needing a tokenizer. The model splits input into patches and pairs a large global transformer with smaller local submodels, which brings self-attention cost down to sub-quadratic and frees compute for much larger feedforward layers. The author generalized the design beyond the paper, so you can stack more than two levels of global and local models, which makes the code a useful starting point for researchers experimenting with long-context or tokenization-free modeling. It installs with a single pip command and exposes a straightforward API for training, loss computation, and sampling, plus a character-level enwik8 training script to verify the setup. This is a research building block rather than a turnkey application, so plan to bring your own data and training loop, but it is a solid reference for anyone exploring byte-level and long-sequence transformer designs.