performer-pytorch

Why we included this project

Machine learning engineers and researchers who need to train transformers on very long sequences will find a practical PyTorch implementation of the Performer architecture here. The library packages the FAVOR+ random-feature linear attention mechanism into ready-made modules: a language model, encoder-decoder pairs, and standalone self-attention and cross-attention layers. That removes the need to reimplement the math from the paper when you want a larger context window. It also combines efficiency tricks from related work, such as reversible layers, local attention heads, GLU feedforwards, and rotary positional embeddings, which makes it a solid base for long-context experiments. Because it ships plain building blocks rather than pretrained weights, it doubles as a useful way to see how linear attention is actually put together.