linear-attention-transformer

Why we included this project

This is a compact PyTorch implementation of a transformer that swaps the usual quadratic attention for a linear-complexity variant, which helps when you train or fine-tune language models on very long sequences where memory and compute costs grow quickly. It mixes local (QKᵀ)V attention with global Q(KᵀV) attention, so the model keeps a receptive field over the whole sequence without paying the full O(n²) cost. The pip-installable package ships ready-to-use language-model and plain-transformer classes, plus knobs for reversible layers, feedforward chunking, GLU variants, and axial attention. It is a research-oriented codebase rather than a maintained production framework, but that makes it a useful reference for engineers and researchers prototyping efficient long-context architectures or benchmarking linear attention against standard transformers.