ChatGLM-finetune-LoRA

Why we included this project

ChatGLM-6b is a capable Chinese-language chat model, but fine-tuning the whole thing is expensive. This project shows how to adapt it with low-rank adaptation (LoRA), which keeps only a small fraction of parameters trainable and brings memory needs down to about 24GB, so a single RTX 3090 can handle training. The code covers the whole pipeline: inserting LoRA layers, building prompt-completion datasets, and launching runs through Hugging Face accelerate, with multi-GPU and DeepSpeed support. It also includes fine-tuned weights trained on the Stanford Alpaca dataset, handy for checking what output to expect before you run your own job. If you want a concrete, copyable starting point for instruction-tuning a ChatGLM-class model, the training and save/load examples are easy to follow.