14,575 native Arabic multiple-choice exam questions across 40 subjects, sourced from school and professional tests in eight countries rather than translated from English MMLU.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | native Modern Standard Arabic multitask exam QA (40 subjects, not a translation of English MMLU) |
| Page status | active |
| Metric | accuracy (paper); exact_match (HELM) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 14575 |
| Dataset licence | The GitHub README states Creative Commons Attribution-NonCommercial-ShareAlike 4.0; the Hugging Face card tag is cc-by-nc-4.0. The paper PDF does not state a licence. This page leaves the SPDX id unset rather than picking one of those two readings. |
| Publisher | Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), with Prince Sattam bin Abdulaziz University, KFUPM, Core42, NYU Abu Dhabi and the University of Melbourne |
ArabicMMLU tests whether a model can answer multiple-choice questions written in Modern Standard Arabic and drawn from real school, university and professional exams across North Africa, the Levant and the Gulf. The 40 subjects span STEM, social science, humanities, Arabic language, and an "other" group that includes driving tests and general knowledge. The authors built it because Arabic LLM papers had been scoring models on English MMLU translated into Arabic, which cannot test Arabic-specific history, law, civics or driving content. Over half the items are described as tailored to Arabic-speaking contexts.
Multiple-choice question answering in Modern Standard Arabic, with 2 to 5 options and one correct answer. Some items, especially Arabic Language (General), include a reading-passage Context field. The original paper scores open models by first-token letter probability (A-E as needed) and closed models by a regex on the first generated token. HELM instead uses joint multiple-choice generation with Arabic instruction text and Arabic option letters (أ ب ج د هـ), scored by exact_match.
No model card in ModelSpec reports this benchmark yet.