HELM scenario that scores a small InteractiveQA subset of five MMLU subjects as four-choice exact match, not the full 57-subject test.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | HELM five-subject MMLU slice with a CodaLab InteractiveQA test subset |
| Page status | unknown |
| Metric | exact_match |
| Direction | higher_is_better |
| Unit | % |
| Dataset licence | MIT |
| Publisher | Stanford CRFM (HELM) |
InteractiveQA MMLU is still four-choice academic questions from Hendrycks et al.'s MMLU, but HELM does not score the full 14k-item test. The scenario class InteractiveQAMMLUScenario subclasses MMLUScenario and keeps only college_chemistry, global_facts, miscellaneous, nutrition and us_foreign_policy. Test CSVs are unpacked from a CodaLab bundle labeled InteractiveQA, not from the ordinary MMLU test folder. Dev CSVs still come from the standard MMLU download and are used as the HELM train split.
Four-option multiple choice. HELM Classic run spec interactive_qa_mmlu uses adapter multiple_choice_joint, instructions "The following are multiple choice questions (with answers) about {subject}", input noun Question, output noun Answer, and exact-match metrics. get_multiple_choice_adapter_spec defaults to max_train_instances=5 and max_tokens=1. The run name is interactive_qa_mmlu:subject={subject}; groups include mmlu.
No model card in ModelSpec reports this benchmark yet.