MMMU

Why we included this project

MMMU is a demanding public benchmark for anyone training or fine-tuning multimodal models who wants to know whether the model actually reasons rather than pattern-matches. Its roughly 11,500 college-level questions come from exams, quizzes, and textbooks across six disciplines, and the images span everything from charts and diagrams to music sheets and chemical structures, so it tests perception and domain knowledge together. The repo includes the evaluation code and, since early 2026, the test-set answers, which means you can score your own model locally instead of depending on a hosted leaderboard. The MMMU-Pro companion adds harder multiple-choice items and vision-only settings for a stricter read on genuine understanding. For teams comparing open-weight vision-language models, this is a practical, reproducible yardstick.