LongNet

Why we included this project

Transformer training usually grinds to a halt on long inputs because attention cost grows quadratically with sequence length. LongNet is a PyTorch implementation of the dilated attention mechanism from the LongNet paper, which spreads the attention field exponentially as token distance grows, keeping compute roughly linear even on sequences far beyond what standard self-attention can handle. You get a standalone DilatedAttention module you can drop into an existing model, plus a ready-to-train LongNetTransformer built from feedforward, layernorm, and SwiGLU blocks, with a training script for enwiki8. It is a research building block rather than a finished product, so plan to adapt it to your own architecture instead of running it as-is. If you are experimenting with very long context windows or document-scale sequences, this is a solid reference to study and extend.