alpaca_eval

Why we included this project

AlpacaEval is for people who tune instruction-following models and want to know, fast and cheaply, whether the latest checkpoint is actually better than the last one. You feed it your model's outputs, it compares them against a reference model, and a strong annotator (GPT-4 by default) scores win rates on a fixed instruction set, so you can run many comparisons during development without paying human raters each time. The length-controlled win rates fix the quirk where long, wordy answers inflated scores, and the results track the much larger Chatbot Arena crowdsourced rankings closely. There is also a leaderboard, precomputed model outputs, human preference data, and tooling for building your own evaluators if you want to change how the annotation works. Treat it as a fast proxy for human judgment rather than a substitute for careful release decisions on high-stakes tasks.