A harder MMMU variant that filters out text-answerable questions, expands options to ten, and adds a vision-only setting where the question is embedded in a photo or screenshot.
unassessed
| Category | multimodal |
|---|---|
| Subcategory | expert multi-discipline knowledge, filtered against text-only shortcuts, with a vision-only setting |
| Page status | active |
| Metric | accuracy, averaged across the Standard (10-option) and Vision settings |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 3460 |
| Dataset licence | Apache-2.0 |
| Publisher | Carnegie Mellon University (MMMU Team, a multi-institution collaboration) |
MMMU-Pro rebuilds MMMU specifically to close two shortcuts the authors found models exploiting: answering from the text alone, and guessing from patterns among the answer options. Construction ran in three steps. First, four strong text-only open-source LLMs each answered MMMU questions without seeing the images, ten times per question; any question at least three of the four models answered correctly across most trials was excluded, and 1,800 questions were then sampled evenly across MMMU's 30 subjects (60 each) from what remained. Second, candidate answer options were expanded from four to ten (GPT-4o generates, Claude 3.5 Sonnet filters, two rounds of human review refine), which itself removed 70 more low-quality questions, leaving 1,730. Third, a vision-only setting embeds each question and its options inside a photograph or screenshot -- captured by human annotators across varied backgrounds, fonts and display conditions -- so a model must read the question rather than receive it as separate text, mirroring how people often share a question by screenshotting it.
Multiple-choice questions paired with one or more images, evaluated in three settings: Standard (4 options, the original unaugmented MMMU-style question, reported only as a difficulty-comparison baseline and excluded from the headline score), Standard (10 options, the augmented format), and Vision (the same 10-option question rendered as a single photo or screenshot with no separate text input). The official MMMU-Pro score averages the Standard-10-option and Vision settings only. Both Direct and chain-of-thought (CoT) prompting are evaluated, with the paper reporting whichever is higher; the paper separately finds OCR-specific prompting has little effect while CoT generally helps.
No model card in ModelSpec reports this benchmark yet.