LoRA

Why we included this project

LoRA is the original reference implementation of the low-rank adaptation technique that has become a standard way to fine-tune large language models without retraining every parameter. It freezes the original weights and learns pairs of small rank-decomposition matrices, so you train a fraction of the model while keeping the same inference latency, and the repo shows the trade-offs directly with benchmark tables for RoBERTa and DeBERTa on GLUE. Ready-to-run examples cover GPT-2 for generation and the GLUE tasks, with checkpoint downloads included. The loralib package is compact and readable, which makes it a practical starting point if you want to understand the method or adapt it to your own PyTorch models. Teams that prefer a maintained, broadly compatible solution can turn to the Hugging Face PEFT library, which now supports LoRA; this repository remains the clearest look at the original design and experimental results.