A 14-category, four-option knowledge test with a closed 10k test set, built so MMLU-style leakage is harder.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | contamination-controlled four-option multitask knowledge (14 categories) |
| Page status | active |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 10000 |
| Dataset licence | CDLA-Permissive-2.0 (dataset card); GitHub badge lists MIT for code |
| Publisher | Microsoft |
MMLU-CF gives a four-option English multiple-choice question from one of 14 broad categories (Math, Physics, Chemistry, Law, Engineering, Other, Economics, Health, Psychology, Business, Biology, Philosophy, Computer_Science, History). The model must reply A, B, C or D. The authors built a new item pool, then applied three decontamination rules, including randomly replacing one distractor with "None of the other choices." The official test set stays closed; the public 10k validation set is what OpenCompass actually scores.
Four-option MCQ, generative letter extraction (OpenCompass first_option_postprocess on ABCD). Publisher protocol: 0-shot and 5-shot. OpenCompass configs mmlu_cf_few_shot and mmlu_cf_zero_shot use FixKRetriever ids 0–4 from the 5-row per-category dev split, or ZeroRetriever. They load microsoft/MMLU-CF and treat each category's val split as test.
No model card in ModelSpec reports this benchmark yet.