judgeval

Why we included this project

Most teams building LLM agents learn about failures only after a customer runs into one, by which point the trace that explains it is usually gone. Judgeval addresses that gap by pairing OpenTelemetry-based tracing with judge-style evaluation in a single Python SDK. It captures inputs, outputs, and token usage automatically, then scores agent behavior against prompt-defined rubrics that return structured, labeled signals. The payoff is the loop this enables: you can run those judges against live production traffic or replay them over historical traces, so fixes get validated against real cases before they ship. Auto-instrumentation for OpenAI, Anthropic, and LangGraph keeps the setup cost low.