evalscope

Why we included this project

Deciding which LLM or multimodal model actually ships is usually a judgment call, and EvalScope makes it less guesswork by running candidates through established benchmarks like MMLU, GSM8K, C-Eval, and AIME with a single command, across text, vision, embedding, reranker, and AIGC models. Accuracy scores are paired with inference stress testing, reporting metrics like TTFT and TPOT, and a WebUI where you can compare models side by side and drill into individual predictions. Teams shipping agentic or tool-using systems get a controlled multi-turn agent loop with Docker sandboxing, and every turn's trace is recorded so you can see exactly where a model stumbled. Since it integrates backends such as OpenCompass and VLMEvalKit, you can reuse harnesses you already know instead of rebuilding. The Arena pairwise battle feature ranks models by preference rather than a single aggregate score, which often matches how real deployment decisions get made.