AgentBench

Why we included this project

AgentBench is a benchmark built for teams that need to judge how well a language model performs as an autonomous agent, not just how fluently it chats. It runs models through eight distinct environments, including operating-system shells, databases, knowledge graphs, a card game, lateral-thinking puzzles, and shopping and browsing tasks, and scores them on whether they achieve concrete goals. That gives you a realistic read on where a model struggles, which is useful when you're deciding between a general-purpose model and one tuned for a specific job. The current release also includes containerized environments and a Docker Compose setup, so you can reproduce an evaluation run on your own hardware without wiring each task by hand. For anyone building agent products, treating this as a repeatable harness beats relying on demos.