harbor

Why we included this project

Anyone who has wired an agent into a benchmark harness knows most of that work is glue code rather than model science. Harbor is a CLI framework from the Terminal-Bench team that condenses the setup into a single command: name an agent like Claude Code or OpenHands, a dataset such as Terminal-Bench-2.0 or SWE-Bench, and a compute target, and it runs the evaluation hundreds of times in parallel, locally on Docker or spread across cloud sandboxes from providers like Daytona and Modal. Teams get comparable, reproducible numbers on how a model or prompt change actually moves task performance, and beyond plain evals the tool can generate rollout traces for reinforcement learning and fine-tuning. For anyone building or shipping agents, it turns scattered ad-hoc testing into a repeatable measurement workflow.