LongLoRA

Why we included this project

Fine-tuning an LLM to handle much longer contexts usually demands training on long sequences at full cost, which blows up compute and memory budgets fast. LongLoRA sidesteps that by combining LoRA-style low-rank updates with a shift sparse attention pattern, so you can stretch effective context to tens of thousands of tokens while updating only a fraction of the parameters. The repo is a complete, reproducible package: training scripts, the LongAlpaca-12k instruction dataset, and checkpoints from 7B up to 70B covering 8k to 100k contexts, plus QLoRA support to trim GPU memory and StreamingLLM inference for long multi-turn conversations. If you already build on Hugging Face and want to try longer-context fine-tuning on your own data, this is a well-documented place to start.