reformer-pytorch

Why we included this project

The Reformer architecture made long-context transformers practical by replacing exact attention with a locality-sensitive hashing approximation, and this library is a solid way to actually use it in PyTorch. You get the full language model plus the reversible encoder and LSH attention as separate classes, so you can pull out just the piece you need rather than reimplementing the paper. That split also makes it a decent reference for seeing how hashing, reversible layers, and chunking fit together. The knobs are extensive, from rotary and axial embeddings to persistent memory keys and masking, which helps when you want to probe the design space. If your work involves 8k to 16k token contexts on a modest GPU, this is worth evaluating before you build from scratch.