llm-as-a-verifier

Why we included this project

LLM-as-a-Verifier is a Python library that scores how well an agent's output actually solves a task, rather than returning a simple pass or fail verdict. The scores come from averaging over the full logprob distribution of the model's score tokens, which yields feedback fine-grained enough to catch small errors a binary check would miss. You can feed that signal into best-of-N selection, test-time scaling, or reinforcement learning, and it runs against DeepSeek, Vertex, or any OpenAI-compatible server that returns logprobs, so it drops into an existing stack without retraining your own model. Teams building coding, robotics, or medical agents will find it useful for choosing among several candidate outputs and for catching corrections a single boolean check would gloss over. If you have ever wanted an agent to tell you why it got an answer wrong instead of only that it failed, this is a reasonable place to start.