evaluation-guidebook

Why we included this project

Deciding how to test an LLM is often harder than building it, and this guidebook collects the lessons the authors learned running the Open LLM Leaderboard and building lighteval. It covers automatic benchmarks, human evaluation, and LLM-as-a-judge setups, then goes on to troubleshooting and reproducibility notes that read like they were written after real deployment struggles. Beginners can start with the basics in each chapter, while teams already doing evals can jump straight to the design and tuning sections, and the starred links point to deeper reading the authors actually recommend. It is a solid reference for anyone writing custom evals for a production model or trying to pick sensible metrics before launch. One caveat: the repository is no longer maintained and redirects to a newer hosted version, so treat it as the stable core of the material.