A grade-school science multiple-choice question set split into Easy and Challenge halves, whose scores are reported interchangeably far too often despite very different difficulty.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | grade-school science multiple-choice QA (Easy and Challenge splits) |
| Page status | superseded |
| Metric | accuracy (often reported as acc_norm, length-normalised) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 7787 |
| Dataset licence | CC-BY-SA-4.0 |
| Publisher | Allen Institute for AI (AI2) |
The AI2 Reasoning Challenge (ARC) is a set of 7,787 natural, grade-school-level science exam questions, split by the original authors into two pools of different difficulty: ARC-Easy (5,197 questions) and ARC-Challenge (2,590 questions). A question lands in the Challenge set only if two baseline solvers of the time -- an information-retrieval solver and a word-co-occurrence (PMI) solver -- both answered it incorrectly; anything either solver could get right went into Easy. That single filtering rule is the entire distinction between the two splits: they share the same format, the same source, and the same authors, and differ only in whether simple lexical-matching methods could solve them. Because both splits are commonly reported under the bare name "ARC" without specifying which one, and a model's score on Easy can be 20-30 points higher than on Challenge, conflating the two is one of the more common reporting errors in this space.
Multiple-choice science question, typically 4 answer options, single correct answer; identical format for both splits.
No model card in ModelSpec reports this benchmark yet.