LWM

Why we included this project

Teams pushing the limits of long context windows will find LWM a genuinely useful codebase. It is a 7B-parameter multimodal autoregressive model trained on long videos and books, able to understand and generate text, images, and video with a context window of up to a million tokens, which puts it in the same territory as long-document retrieval and long-video understanding. The repo goes well beyond the paper: the training stack is heavily optimized around RingAttention, masked sequence packing for mixed sequence lengths, and loss weighting that balances language and vision objectives, so it works both as a codebase for training on long sequences yourself and as a source of ideas for taming memory and compute in multimodal training. The authors also release text-only and chat checkpoints, so you can grab a ready model for long-document QA or long-video conversation without training anything from scratch.