A re-annotated slice of MMLU (5,700 items across 57 subjects in the 2.0 release) that tags label errors instead of adding new questions.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | manually re-annotated MMLU subset (label-error audit) |
| Page status | active |
| Metric | accuracy (exact match on A–D) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 5700 |
| Dataset licence | CC-BY-4.0 |
| Publisher | University of Edinburgh (with Sapienza, University of Bari, University College London and co-authors) |
MMLU-Redux keeps Hendrycks MMLU questions and four options, then has experts tag each item with an error_type (including "ok") and, when needed, a corrected answer. The evaluated skill is still four-option academic/professional knowledge, but the score is meant to be read on items the annotators judged valid. English text. It is not a new exam like [MMLU-Pro](mmlu_pro.md) or the contamination-controlled [mmlu_cf](mmlu_cf.md).
Four-option MCQ. lm-evaluation-harness group `mmlu_redux_generative` uses generate_until with regex `([ABCD])`, exact_match, dataset_path `fxmarty/mmlu-redux-2.0-ok`. Version 4 (PR 3410) keeps rows with error_type="ok" and drops 370 of 5,700 (~6.5%). Per-subject tasks are named `mmlu_redux_<subject>_generative`. An earlier Hugging Face dump, edinburgh-dawg/mmlu-redux, is 30 subjects × 100 = 3,000 rows from the first paper draft.
No model card in ModelSpec reports this benchmark yet.