{
 "body": "\n## What it measures\n\nMMLU-SR tests whether a model can preserve the meaning of an MMLU question when a key term is replaced by a made-up symbol. The prompt supplies a definition for the symbol. The benchmark covers the same broad academic subjects as MMLU, including STEM, humanities, social sciences, and professional topics.\n\nIt has three variants: replacement in the question, replacement in answer choices, and replacement in both. This changes terminology while retaining the underlying question structure. It is a reasoning stress test, not a new subject exam.\n\n## How it is scored\n\nThe harness reports multiple-choice accuracy. Each item has four choices, giving a 25% uniform-choice baseline. Aggregate `mmlusr` contains the three replacement settings, while `mmlusr_question_only`, `mmlusr_answer_only`, and subject tasks expose narrower slices.\n\nThe paper also discusses fine-tuning and a few-shot setup. Those results are not interchangeable with a zero-shot harness run. Report the replacement variant, subject aggregation, and shot count with every score.\n\n## Dataset and licence\n\nThe official Hub card reports 42,639 rows across the three configurations. Each configuration has development/train and test data covering 57 subjects. The Hub metadata labels the dataset MIT. Questions are derived from MMLU and the public test files include answer labels.\n\n## Who publishes it\n\nWentian Wang, Sarthak Jain, Paul Kantor, Jacob Feldman, Lazaros Gallos, and Hao Wang introduced MMLU-SR in 2024. The authors maintain the GitHub release, while EleutherAI maintains the lm-evaluation-harness task group.\n\n## Lineage\n\nMMLU-SR is a variant of [MMLU](mmlu.md), not a subject page. Its three replacement modes are variants of one benchmark. It is distinct from [MMLU-Pro](mmlu_pro.md), [MMLU-Pro+](mmlu_pro_plus.md), and MMLU-Redux.\n\n## Saturation and contamination\n\nThe paper reports large performance reductions after replacing terminology, leaving room for separation. A current frontier saturation value is not established. Underlying MMLU questions and MMLU-SR test files are public, so contamination risk is high even though the dummy symbols alter surface form.\n\n## How to run it\n\nUse `lm_eval --tasks mmlusr` or a named subject/variant task. Match the harness version because task files and few-shot handling can change. The original repository also documents a fine-tuned evaluation, which should be reported separately from harness in-context evaluation.\n\n## Reading the numbers\n\nA high score means the model selected the answer after applying the supplied replacement definitions. A drop relative to ordinary MMLU can indicate reliance on familiar terminology, but it can also reflect awkward definitions or prompt formatting. Compare all three replacement modes with the corresponding MMLU subjects under the same evaluation budget.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mmlusr",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "MMLU-SR",
   "MMLU Symbol Replacement",
   "MMLU-R"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "MMLU-SR modifies public MMLU questions and publishes its test files. The symbol substitutions reduce exact-string overlap but do not establish low contamination.",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 42639,
   "size_note": "The Hugging Face repository reports 42,639 rows across three variants, each with development/train and test data over 57 subjects.",
   "splits": "question_only, answer_only, and question_and_answer variants; each has train/dev and test files",
   "url": "https://huggingface.co/datasets/NiniCat/MMLU-SR"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna, luna-batch-059 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, luna-batch-059"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "mmlusr",
   "opencompass": "",
   "other": ""
  },
  "id": "mmlusr",
  "last_updated": "2025-05",
  "leaderboard_url": "",
  "lineage": {
   "family": "mmlu",
   "predecessor": "mmlu",
   "successors": [],
   "variants": []
  },
  "measures": "MMLU-SR asks the model to answer familiar MMLU questions after replacing key terms with invented symbols and definitions. It has question-only, answer-only, and question-and-answer variants across the 57 MMLU subjects. The task stays text-only and multiple choice while changing terminology.\n",
  "metric": {
   "baseline_note": "The public tasks use four answer choices, so uniform-choice chance is 25%; no human score was established.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy (acc)",
   "random_baseline": 25.0,
   "unit": "%"
  },
  "name": "MMLU-SR",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2406.15468",
   "title": "MMLU-SR: A Benchmark for Stress-Testing Reasoning Capability of Large Language Models",
   "url": "https://arxiv.org/abs/2406.15468",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Wentian Wang",
    "Sarthak Jain",
    "Paul Kantor",
    "Jacob Feldman",
    "Lazaros Gallos",
    "Hao Wang"
   ],
   "org": "MMLU-SR authors",
   "url": "https://github.com/Wang-ML-Lab/MMLU-SR"
  },
  "released": "2024-06",
  "repo_url": "https://github.com/Wang-ML-Lab/MMLU-SR",
  "saturation": {
   "as_of": "",
   "note": "The paper reports substantial drops after symbol replacement; no current leaderboard top score was established.",
   "status": "open",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "MMLU-SR paper HTML",
    "url": "https://arxiv.org/html/2406.15468"
   },
   {
    "accessed": "2026-09-08",
    "title": "Official MMLU-SR dataset card",
    "url": "https://huggingface.co/datasets/NiniCat/MMLU-SR"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness MMLU-SR README",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/mmlusr/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "Official MMLU-SR repository",
    "url": "https://github.com/Wang-ML-Lab/MMLU-SR"
   }
  ],
  "status": "active",
  "subcategory": "symbol-replacement stress test over 57 MMLU subjects",
  "summary": "An MMLU variant that replaces key terms with defined dummy symbols to separate conceptual reasoning from surface pattern matching.",
  "tags": [
   "mmlu",
   "reasoning",
   "symbol-replacement",
   "multiple-choice"
  ],
  "task_format": "Four-option multiple choice. The replaced term is paired with a definition in the prompt. The lm-evaluation-harness exposes aggregate, question-only, answer-only, and subject tasks; the original paper also describes a fine-tuning evaluation.\n"
 }
}