local-attention

Why we included this project

Full self-attention is usually the bottleneck in a long-context transformer, since the cost grows quadratically with sequence length. This library takes the practical route of windowed attention: each position attends only to tokens within a configurable window, with an option to also look back at the previous window. That keeps memory and compute costs much lower while keeping language-modeling quality strong, and the author has already used the same code in several other sparse-attention projects. It is a plain PyTorch module that drops into an existing training loop: you pick the window size, decide whether attention is causal, and flags handle lookback, shared query/key space for Reformer-style models, and automatic padding. A LocalTransformer wrapper shows how the windowed layer slots into a complete transformer stack, which makes it a handy reference even if you end up wiring the pieces yourself.