SPIN

Why we included this project

SPIN takes a model that has already been instruction-tuned and makes it stronger by having it play against its own earlier versions. The model generates its own responses and learns to tell them apart from the original supervised fine-tuning examples, so each round of self-play nudges the policy forward without needing freshly labeled preference data. That makes it an interesting alternative to DPO for anyone who wants to lift a fine-tuned model's benchmark and MT-Bench scores without curating more human preference pairs. The repo includes the generation scripts, the fine-tuning loop, and the Hugging Face datasets and zephyr-7b checkpoints used at each iteration, so results are reproducible, and the two-step generate-then-finetune workflow can be adapted to a different base model.