evals

Why we included this project

Teams shipping products on top of LLMs often struggle to tell whether a new model version actually helps their use case. Evals gives you a way to turn that worry into repeatable tests: it includes a registry of ready-made benchmarks covering different model behaviors, and you can write your own evals using your own data, including private ones that never leave your environment. Many evals are just JSON plus a YAML spec, so you don't need to write evaluation code, which makes it approachable for teams without deep ML expertise. It also supports more advanced setups like prompt chains and tool-using agents. If you've been burned by a silent regression after swapping models, this is a practical way to catch those changes before they reach users.