MMLU-Pro

A harder, ten-option successor to MMLU with about 12,000 reasoning-heavy questions across 14 categories, built to restore headroom lost to MMLU's saturation at the frontier.

unassessed

What it measures

MMLU-Pro gives a model a question and up to ten labelled answer options (most questions use all ten; a small number carry fewer after manual review removed unreasonable distractors), drawn from 14 broad categories spanning STEM, humanities, social sciences and business/health/other topics: Biology, Business, Chemistry, Computer Science, Economics, Engineering, Health, History, Law, Math, Philosophy, Physics, Psychology and Other. The model must select the single correct option. Roughly 57% of the questions are a difficulty-filtered subset of the original MMLU test set (with "trivial and ambiguous" items removed); the remaining 43% are newly written from a STEM question website, TheoremQA and SciBench, with GPT-4-generated distractors expanding every question from four options toward ten, reviewed afterward by a panel of over ten subject-matter experts.

Task format

Multiple-choice question answering with up to ten labelled options, graded on the single option the model selects. The reference protocol is 5-shot with chain-of-thought prompting, with fewshot examples drawn from a dedicated 70-question validation split; scoring extracts the answer letter from free-form generated text (for example via a regex matching "the answer is (X)") rather than comparing option log-likelihoods, because the authors found direct/log-likelihood scoring under-performs chain-of-thought by up to 19 points on this dataset -- the opposite of the original MMLU. Across 24 prompt styles the authors tested, score sensitivity to prompt wording fell from 4-5% on MMLU to about 2% on MMLU-Pro.

Models reporting this benchmark

Showing the top 200 of 354.

Data

This page as JSON · Edit on GitHub