evals-skills

Why we included this project

LLM evaluation pipelines tend to fail in predictable ways, and this plugin encodes the mistakes its author has seen while helping 50+ companies and teaching an AI evals course. It installs as skills for Claude Code or the open Skills CLI, each one walking the agent through a specific job: auditing an existing eval pipeline, reading traces and categorizing failures, generating diverse synthetic test inputs, designing LLM-as-Judge prompts, and calibrating judges against human labels. The validate-evaluator skill is worth a look on its own, since it uses data splits and bias correction to check whether an LLM judge actually agrees with human labels, a step many teams skip. Newcomers can start with eval-audit, which surfaces problems with prioritized severity and points to the other skills that fix them. Because the skills come from mistakes seen across many real companies, they read as practical guardrails rather than generic advice.