long_llama

Why we included this project

LongLLaMA is built on OpenLLaMA and trained with the Focused Transformer method, which stretches usable context length well beyond the usual few-thousand-token window. The repo ships a base 3B model, an instruct-tuned 3B, and code-focused 7B checkpoints on Hugging Face, plus the training code for continued pretraining and instruction fine-tuning, so it works both as a ready-to-load model and as a reference for reproducing context-scaling techniques. That combination matters if you work with long documents, want to summarize or reason over large text, or have code that references big codebases, since the provided Colab notebooks and inference paths give you a practical starting point without much setup.