titans-pytorch

Why we included this project

Transformers run into trouble when the context window gets long, and the Titans line of work attacks that by adding a neural memory module that keeps learning at test time. This repo is the unofficial PyTorch port, and it ships the pieces ready to use: a standalone NeuralMemory block plus a MemoryAsContextTransformer that pairs local attention with persistent and long-term memory tokens. Both are small, composable, and install with a single pip command, so they drop into an existing training loop without much ceremony. The included training and sampling scripts give you a working baseline for prototyping long-context language models. It is an implementation library, not a pretrained model, so expect to spend real training time before judging how the architecture behaves on your own data.