OpenMythos

Why we included this project

OpenMythos is a PyTorch implementation of the Recurrent-Depth Transformer idea that keeps circulating in frontier-model speculation. Instead of just reading about looped transformers, you get runnable code: a Prelude of transformer blocks, a recurrent block that reuses the same weights across several loops, and a Coda, with attention switchable between MLA and GQA and a sparse MoE feed-forward. The authors are upfront that this is a theoretical reconstruction from public research, not a pretrained model, so treat it as a study aid and sandbox rather than production software. It scales from 1B to 1T parameters, and the included training script lets you test compute-adaptive, depth-variable reasoning on your own hardware. If you want to see how recurrence inside a single forward pass might actually work, this is a concrete place to start.