RETRO-pytorch

Why we included this project

RETRO-pytorch turns DeepMind's RETRO paper into something you can actually run. The model queries an index built over your own corpus and conditions each generated chunk on neighboring text it retrieves, so generation leans on an external datastore instead of memorizing everything in weights. The fiddly parts are handled for you, including chunked causal cross-attention, rotary position embeddings, and deep-norm scaling for very deep stacks, with Faiss and autofaiss doing the index construction and nearest-neighbor search. Training wrappers and dataset helpers also convert a folder of plain text into the memory-mapped arrays and neighbor references training expects. RETRO's pitch is reaching GPT-3 performance with 10x fewer parameters, and if you want to test that claim on your own data, this is a concrete place to start.