{
 "body": "\n## What it measures\n\nInteractiveQA MMLU still asks four-choice academic questions, but only on five MMLU subjects: college chemistry, global facts, miscellaneous, nutrition and US foreign policy. HELM loads ordinary MMLU dev items as in-context examples and a CodaLab \"InteractiveQA\" unpack as the test set. The scenario comment calls that unpack a small subset of the original test. This is not the 57-subject [MMLU](mmlu.md) mean, and it is not the per-subject pages [mmlu_college_chemistry](mmlu_college_chemistry.md), [mmlu_global_facts](mmlu_global_facts.md), [mmlu_miscellaneous](mmlu_miscellaneous.md), [mmlu_nutrition](mmlu_nutrition.md) or [mmlu_us_foreign_policy](mmlu_us_foreign_policy.md) when those score the full Hendrycks test CSV.\n\nThe name InteractiveQA is the HELM suite label. The Python scenario still grades static multiple choice. No multi-turn human dialogue is implemented in the file opened for this page.\n\n## How it is scored\n\nHELM Classic uses joint multiple-choice prompting and exact match. Default few-shot is five dev items. Chance is 25% on four options. A `helm-run --max-eval-instances` cap is a runner setting, not a property of the CSVs. Do not compare a truncated InteractiveQA cell to a full-subject MMLU number.\n\n## Dataset and licence\n\nMMLU is MIT. HELM's harness is Apache-2.0. This page records MIT for the questions. How many test rows sit in the CodaLab tarball was not counted here. Answers in that tarball are still the public MMLU keys.\n\n## Who publishes it\n\nStanford CRFM ships the scenario in HELM Classic (`classic_run_specs.py`). The HELM paper is Liang et al., arXiv:2211.09110 (November 2022). A historic results UI remains at nlp.stanford.edu/helm/interactive_qa_mmlu/. The HELM README states the project entered maintenance mode on 1 June 2026.\n\n## Lineage\n\nPredecessor: [MMLU](mmlu.md). This is a HELM protocol and a five-subject slice, not a new item set. It is not MMLU-Pro, MMLU-Redux or a language transfer. Do not treat `interactive_qa_mmlu` as a harness spelling of `mmlu`.\n\n## Saturation and contamination\n\nFull MMLU is saturated at the frontier; this slice has no confirmed top score. The questions have been public since 2020. A smaller test file does not make them private.\n\n## How to run it\n\n```\nhelm-run --run-entries interactive_qa_mmlu:subject=nutrition,model=<id> --suite <suite>\n```\n\nSubject must be one of the five names above. Passing `anatomy` or another MMLU subject raises an assertion in the scenario. Quote subject, shot count and `max_eval_instances` when comparing.\n\n## Reading the numbers\n\nA high exact-match here means the model picked the labeled letter on HELM's InteractiveQA rows for that subject. It does not stand in for 57-subject MMLU, MMLU-Pro or a human-in-the-loop study. Short runner caps bounce. Read it beside the matching `mmlu_*` subject page, which uses the full Hendrycks test CSV, and treat disagreement as a protocol difference rather than a model ranking.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "interactive_qa_mmlu",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "interactive_qa_mmlu",
   "HELM InteractiveQA MMLU"
  ],
  "category": "knowledge",
  "contamination": {
   "note": "The questions are public MMLU items (MIT-licensed, on GitHub and Hugging Face since 2020). Using a smaller CodaLab slice does not hold out answers from pretraining. This is from publicity and age, not a measured leakage study of the CodaLab files.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "Item count for the CodaLab InteractiveQA test unpack was not established. The scenario comment says \"a small subset of the original test set.\" The live nlp.stanford.edu InteractiveQA MMLU page did not yield a row count or a max_eval_instances figure in this review. Dev/train still uses the standard MMLU per-subject dev CSV.\n",
   "splits": "HELM train from MMLU dev CSV; HELM test from CodaLab InteractiveQA unpack",
   "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/interactive_qa_mmlu_scenario.py"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-050 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-050"
  },
  "harness": {
   "bigbench": "",
   "helm": "interactive_qa_mmlu",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "helm-run --run-entries interactive_qa_mmlu:subject=<one of five>,model=<id>. Valid subjects are hardcoded. Do not pass an ordinary MMLU subject.\n"
  },
  "id": "interactive_qa_mmlu",
  "last_updated": "",
  "leaderboard_url": "https://nlp.stanford.edu/helm/interactive_qa_mmlu/",
  "lineage": {
   "family": "",
   "predecessor": "mmlu",
   "successors": [],
   "variants": []
  },
  "measures": "InteractiveQA MMLU is still four-choice academic questions from Hendrycks et al.'s MMLU, but HELM does not score the full 14k-item test. The scenario class InteractiveQAMMLUScenario subclasses MMLUScenario and keeps only college_chemistry, global_facts, miscellaneous, nutrition and us_foreign_policy. Test CSVs are unpacked from a CodaLab bundle labeled InteractiveQA, not from the ordinary MMLU test folder. Dev CSVs still come from the standard MMLU download and are used as the HELM train split.\n",
  "metric": {
   "baseline_note": "Four options, so uniform chance is 25%. Hendrycks et al. estimate expert- level accuracy on the full 57-subject MMLU test at about 89.8%; that figure does not apply to this five-subject InteractiveQA slice. No human figure was read for the CodaLab subset.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "exact_match",
   "random_baseline": 25,
   "unit": "%"
  },
  "name": "InteractiveQA MMLU",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2211.09110",
   "title": "Holistic Evaluation of Language Models",
   "url": "https://arxiv.org/abs/2211.09110",
   "year": 2022
  },
  "publisher": {
   "authors": [],
   "org": "Stanford CRFM (HELM)",
   "url": "https://crfm.stanford.edu/helm/classic/"
  },
  "released": "2022-11",
  "repo_url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/interactive_qa_mmlu_scenario.py",
  "saturation": {
   "as_of": "",
   "note": "No dated top exact-match for this five-subject InteractiveQA slice was read. Full-MMLU saturation is tracked on the [mmlu](mmlu.md) page, not here. HELM entered maintenance mode on 1 June 2026 per the HELM README.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "InteractiveQAMMLUScenario (five subjects, CodaLab test unpack)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/interactive_qa_mmlu_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM Classic get_interactive_qa_mmlu_spec (joint MC, exact match, groups=mmlu)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/run_specs/classic_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "get_multiple_choice_adapter_spec defaults (5-shot, max_tokens=1)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/adaptation/common_adapter_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "Historic InteractiveQA MMLU run page (JS UI; no instance count recovered here)",
    "url": "https://nlp.stanford.edu/helm/interactive_qa_mmlu/?runs=1&runSpec=interactive_qa_mmlu%3Asubject%3Dmiscellaneous%2Cmodel%3Dopenai_text-babbage-001"
   },
   {
    "accessed": "2026-09-08",
    "title": "hendrycks/test MIT License (MMLU questions, copyright 2020 Dan Hendrycks)",
    "url": "https://raw.githubusercontent.com/hendrycks/test/master/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM README (maintenance mode 1 June 2026; Classic leaderboard)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM Apache License 2.0 (harness; MMLU data remain MIT)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "Holistic Evaluation of Language Models (arXiv:2211.09110)",
    "url": "https://arxiv.org/abs/2211.09110"
   },
   {
    "accessed": "2026-09-08",
    "title": "MMLU paper (arXiv:2009.03300)",
    "url": "https://arxiv.org/abs/2009.03300"
   }
  ],
  "status": "unknown",
  "subcategory": "HELM five-subject MMLU slice with a CodaLab InteractiveQA test subset",
  "summary": "HELM scenario that scores a small InteractiveQA subset of five MMLU subjects as four-choice exact match, not the full 57-subject test.",
  "tags": [
   "helm",
   "mmlu",
   "multiple-choice",
   "knowledge",
   "interactive-qa"
  ],
  "task_format": "Four-option multiple choice. HELM Classic run spec interactive_qa_mmlu uses adapter multiple_choice_joint, instructions \"The following are multiple choice questions (with answers) about {subject}\", input noun Question, output noun Answer, and exact-match metrics. get_multiple_choice_adapter_spec defaults to max_train_instances=5 and max_tokens=1. The run name is interactive_qa_mmlu:subject={subject}; groups include mmlu.\n"
 }
}