A 12,032-question MMLU-Pro extension that adds multi-correct pairs to test higher-order reasoning and shortcut resistance.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | multi-correct multiple-choice reasoning across 14 MMLU-Pro subjects |
| Page status | active |
| Metric | accuracy; shortcut selection ratio; correct pair identification ratio |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 12032 |
| Publisher | Autodesk AI Research |
MMLU-Pro+ presents academic multiple-choice questions with up to ten options. It modifies MMLU-Pro questions so some options assert that two answers are both correct. The benchmark tests whether a model can identify valid pairs and resist anchoring on a familiar single answer across 14 subjects.
Multiple choice, usually with ten options. Questions include true-positive pairs, partial false-positive pairs, and complete false-positive pairs. The paper reports accuracy plus shortcut selection ratio and correct pair identification ratio.
No model card in ModelSpec reports this benchmark yet.