OpenJudge

Why we included this project

Anyone building an LLM-powered agent or chatbot eventually hits the same wall: it is easy to tell when the app is obviously broken, but much harder to say whether it is actually good. OpenJudge gives you a repeatable way to answer that. You collect test cases, pick a grader or write a rubric, run the evaluation across your app, and get a breakdown of where it falls short. The built-in graders cover common cases, and you can generate scenario-specific rubrics instead of hand-writing every scoring rule. The grading output can also be converted into reward signals, so the same evaluation you use for QA can feed fine-tuning and RLHF-style training. If you own quality for an LLM application and want one workflow that measures and improves it, this is worth a look.