24,143 real high-school exam questions across 16 languages and 24 subjects (Hardalov et al., 2020); arabic_exams documents the AceGPT-repackaged Arabic slice of this same corpus.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | multilingual high-school exam multiple-choice QA (16 languages, 24 subjects) |
| Page status | active |
| Metric | exact_match (accuracy) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 24143 |
| Dataset licence | CC BY-SA 4.0, per both the GitHub repository's own licence metadata and the Hugging Face dataset card's licence tag |
EXAMS gives a model a real high-school exam question, drawn from actual national school examinations rather than written for the benchmark, with three to five labelled answer options and one correct answer. Its distinguishing feature is breadth: the "multilingual" testbed this page documents spans 16 languages across 8 language families and 24 subjects from the natural and social sciences, so a score on it mixes reading comprehension in a given language with subject-matter recall, and can be read per-language or per-subject rather than only as one blended number. A separate "cross-lingual" testbed (train in one language, test in another) exists in the same release but is not what this page or HELM's `exams_multilingual` scenario evaluate.
Multiple-choice exam question with labelled options (commonly four, though the release's ARC-style format allows a variable number per item) and a single correct answer, in one of 16 languages; HELM parameterises a run by `language` and `subject` (each can be set to "all"), scoring the model's chosen option by exact match.
No model card in ModelSpec reports this benchmark yet.