{
 "body": "\n## What it measures\n\nArabic EXAMS tests whether a model can answer real high-school exam questions written in Arabic. Each\nitem is a multiple-choice question with four labelled options, drawn from actual school examinations\nacross five subjects: Islamic Studies, Biology, Physics, Science, and Social (Studies). Because the\nquestions were written for human students rather than generated for the benchmark, answering them\ncorrectly requires genuine Arabic reading comprehension combined with subject-matter knowledge, rather\nthan any single narrow skill.\n\nThis is the Arabic-language slice of EXAMS, a multilingual and cross-lingual high-school-exam\nquestion-answering dataset introduced by Hardalov et al. (2020). The Arabic subset was later selected,\nsplit by subject, and re-packaged for LLM evaluation as part of the AceGPT project's Arabic evaluation\nsuite, and it is this AceGPT-derived packaging -- not the full multilingual release -- that both harnesses\nin this repository's scope actually load.\n\n## What both harness names load\n\nThis id absorbs two harness entries that turned out to be the same benchmark under different names.\n`aexams` in lm-evaluation-harness loads the dataset `Hennara/aexams` (five per-subject tasks:\n`aexams_Biology`, `aexams_IslamicStudies`, `aexams_Physics`, `aexams_Science`, `aexams_Social`, grouped\nunder `aexams`); its own loading script names its homepage as the AceGPT project's `EXAMS_Arabic`\nevaluation directory. `arabic_exams` in HELM loads `OALL/Arabic_EXAMS`, a mirror the scenario's own\ndocstring describes as \"the Open Arabic LLM Leaderboard (OALL) version mirror of the Arabic subset of\nEXAMS, which is in turn based on the AceGPT version.\" Both therefore trace to the same AceGPT-curated\nsource. This page confirmed the two are identical by unzipping `Hennara/aexams`'s underlying data file\nand counting items directly: 537 test-split questions (35 Biology, 73 Islamic Studies, 42 Physics, 115\nScience, 272 Social) plus 25 dev-split questions (5 per subject) -- 562 total -- which match\n`OALL/Arabic_EXAMS`'s reported 537 test plus 25 validation exactly, and match the original EXAMS-QA\nrepository's own per-language table, which lists 562 total Arabic items. `aexams` is recorded as an\nalias on this page rather than getting a separate page.\n\n## How it is scored\n\nlm-evaluation-harness treats each subject as a loglikelihood-based multiple-choice task and reports\naccuracy and length-normalised accuracy (acc_norm), aggregated as a size-weighted mean across the five\nsubjects under the `aexams` group. HELM instead prompts the model to generate an answer and scores it\nby exact string match (`exact_match`) against the reference letter or text. These are different\nprotocols -- ranking four fixed continuations by likelihood versus scoring a free-form generation -- so a\nmodel's `aexams` score and its `arabic_exams` score are not guaranteed to match even though they\nevaluate the same 562 items. Random guessing among four options scores 25%.\n\n## Dataset and licence\n\n562 questions in total: 537 in the harnesses' test split and 25 in a small dev/validation split, spread\nacross five subjects (Biology, Islamic Studies, Physics, Science, Social). Neither Hugging Face mirror\nused by the two harnesses (`Hennara/aexams`, `OALL/Arabic_EXAMS`) states its own licence on its dataset\ncard. The original EXAMS-QA repository that the questions ultimately come from is released under\nCC BY-SA 4.0, which this subset plausibly inherits, but that has not been confirmed to apply to either\nre-hosted mirror specifically. Both mirrors are text-only and Arabic-only; answers are public in both.\n\n## Who publishes it\n\nThe underlying question set comes from EXAMS, published by Momchil Hardalov, Todor Mihaylov, Dimitrina\nZlatkova, Yoan Dinkov, Ivan Koychev and Preslav Nakov at EMNLP 2020. The specific Arabic, five-subject\nslice used by both harnesses in this repository was selected and split for LLM evaluation by the AceGPT\nproject (Huang et al., \"AceGPT, Localizing Large Language Models in Arabic,\" 2023), whose GitHub\nevaluation directory both Hugging Face mirrors cite as their source. `Hennara/aexams` and\n`OALL/Arabic_EXAMS` are two independent Hugging Face re-hosts of that same AceGPT-curated data, created\nten days apart in February 2024; scores are tracked today on the Open Arabic LLM Leaderboard (OALL)\nspace, which was still receiving updates as of October 2025.\n\n## Lineage\n\nArabic EXAMS is one language slice of the larger multilingual EXAMS corpus (2020), which spans high\nschool exam questions in more languages -- sources reviewed here disagree on the exact count, one giving\n16 languages and another giving 26, a discrepancy not resolved from the sources opened for this page.\nNeither the full multilingual EXAMS benchmark nor any other language slice of it has a page in this\nrepository yet. Within this repository, `aexams` and `arabic_exams` are recorded as the same benchmark\nunder two harness names, per this page's front-matter `aliases`; the fold is also logged in\n`benchmarks/_census/DATA-QUALITY.md`.\n\n## Saturation and contamination\n\nNo dated top-score reading specific to `aexams` or `arabic_exams` was found in the sources reviewed for\nthis page; establishing one would mean pulling per-model results from the OALL leaderboard space, which\nwas not done here. The underlying question-and-answer text has been publicly downloadable since the\noriginal EXAMS release in 2020, and both re-hosted mirrors used by today's harnesses have been public\nsince February 2024, so contamination from broad or Arabic-focused web pretraining is plausible for\nmodels trained since then; no source consulted describes a specific mitigation for this benchmark.\n\n## How to run it\n\nlm-evaluation-harness registers it as the `aexams` group (five tasks: `aexams_Biology`,\n`aexams_IslamicStudies`, `aexams_Physics`, `aexams_Science`, `aexams_Social`), loading\n`Hennara/aexams` with 5-shot loglikelihood scoring by default (`fewshot_split: dev`). HELM registers it\nas the `arabic_exams` scenario, loading `OALL/Arabic_EXAMS` and remapping its `validation` split to\nHELM's train role for in-context examples. Because the two harnesses use different prompt formats,\nfew-shot sources and scoring rules (ranking vs. generation), do not treat an `aexams` score and an\n`arabic_exams` score for the same model as directly comparable without checking which harness produced\neach one.\n\n## Reading the numbers\n\nA high score here says a model can read Arabic well enough to parse a real exam question and recall or\ninfer the right fact among five school subjects; it says little beyond that, since the item count is\nsmall (562) and concentrated in a few subjects, with Social Studies alone making up roughly half the\ntest split. Because this benchmark exists under two names with two different scoring protocols, always\ncheck which harness produced a given number before comparing it across model reports, and treat a bare\n\"Arabic EXAMS\" score with no harness named as unverified until you confirm the source.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "arabic_exams",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "aexams",
   "AEXAMS",
   "EXAMS (Arabic)"
  ],
  "category": "knowledge",
  "contamination": {
   "note": "The underlying EXAMS question set has been publicly downloadable since 2020, and both Hugging Face mirrors used by the two harnesses have been public since February 2024, so a model trained on a broad web or Arabic-web crawl since then has plausibly seen these exact questions and answers.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "ar"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 562,
   "size_note": "562 questions total, confirmed independently from three sources -- the original EXAMS-QA repository's own per-language table (Arabic, 562), the lm-evaluation-harness mirror Hennara/aexams (25 dev + 537 test, unzipped and counted directly), and the HELM mirror OALL/Arabic_EXAMS (25 validation + 537 test, per its Hugging Face dataset card). Split by subject in the test set 35 Biology, 73 Islamic Studies, 42 Physics, 115 Science, 272 Social; 5 per subject in dev/validation.",
   "splits": "lm-evaluation-harness (Hennara/aexams): test (537) / dev (25). HELM (OALL/Arabic_EXAMS): test (537) / validation (25). Same 562 items under different split names.",
   "url": "https://huggingface.co/datasets/OALL/Arabic_EXAMS"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 4, slice D",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "arabic_exams",
   "inspect_evals": "",
   "lm_eval": "aexams",
   "opencompass": "",
   "other": ""
  },
  "id": "arabic_exams",
  "last_updated": "",
  "leaderboard_url": "https://huggingface.co/spaces/OALL/Open-Arabic-LLM-Leaderboard",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "Arabic EXAMS gives a model a high-school-level exam question written in Arabic, with four labelled answer options, and asks it to pick the correct one. The questions are drawn from real school examinations rather than written for the benchmark, and cover five subjects -- Islamic Studies, Biology, Physics, Science, and Social (Studies) -- so a score on this benchmark reflects a mix of Arabic reading comprehension and subject-matter recall rather than any single skill. It is the Arabic-language slice of EXAMS, a multilingual high-school-exam question answering dataset (Hardalov et al., 2020), later re-packaged for LLM evaluation by the AceGPT project.",
  "metric": {
   "baseline_note": "Random baseline assumes four roughly balanced options. lm-evaluation-harness scores each of the five subject tasks with loglikelihood-based multiple-choice accuracy (acc and acc_norm, size-weighted mean across subjects), while HELM's ArabicEXAMSScenario instead asks the model to generate an answer and scores exact_match; the two protocols are not guaranteed to agree for the same model.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100.0,
   "name": "accuracy (lm-evaluation-harness also reports acc_norm; HELM's main metric is exact_match)",
   "random_baseline": 25.0,
   "unit": "%"
  },
  "name": "Arabic EXAMS",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "EXAMS: A Multi-subject High School Examinations Dataset for Cross-lingual and Multilingual Question Answering",
   "url": "https://aclanthology.org/2020.emnlp-main.438/",
   "year": 2020
  },
  "publisher": {
   "authors": [
    "Momchil Hardalov",
    "Todor Mihaylov",
    "Dimitrina Zlatkova",
    "Yoan Dinkov",
    "Ivan Koychev",
    "Preslav Nakov"
   ],
   "org": "",
   "url": "https://github.com/mhardalov/exams-qa"
  },
  "released": "2023",
  "repo_url": "https://github.com/FreedomIntelligence/AceGPT/tree/main/eval/benchmark_eval/benchmarks/EXAMS_Arabic",
  "saturation": {
   "as_of": "",
   "note": "No dated leaderboard reading for this specific task was found in the sources reviewed; the Open Arabic LLM Leaderboard (OALL) space that hosts scores for it was still active as of its last Hugging Face update (October 2025), but a specific top score for arabic_exams/aexams was not confirmed from a source opened for this page.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness aexams task directory (README, group and per-subject yaml configs)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/aexams"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM ArabicEXAMSScenario source",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/arabic_exams_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hennara/aexams dataset (used by lm-evaluation-harness's aexams task)",
    "url": "https://huggingface.co/datasets/Hennara/aexams"
   },
   {
    "accessed": "2026-09-08",
    "title": "OALL/Arabic_EXAMS dataset card and split counts (used by HELM's arabic_exams scenario)",
    "url": "https://huggingface.co/datasets/OALL/Arabic_EXAMS"
   },
   {
    "accessed": "2026-09-08",
    "title": "mhardalov/exams-qa, the original EXAMS dataset repository (README with per-language item counts; CC-BY-SA-4.0 licence on the repository itself)",
    "url": "https://github.com/mhardalov/exams-qa"
   },
   {
    "accessed": "2026-09-08",
    "title": "EXAMS: A Multi-subject High School Examinations Dataset for Cross-lingual and Multilingual Question Answering (ACL Anthology)",
    "url": "https://aclanthology.org/2020.emnlp-main.438/"
   },
   {
    "accessed": "2026-09-08",
    "title": "AceGPT, Localizing Large Language Models in Arabic (curated this Arabic subset for LLM eval)",
    "url": "https://arxiv.org/abs/2309.12053"
   },
   {
    "accessed": "2026-09-08",
    "title": "AceGPT eval benchmark EXAMS_Arabic directory (immediate source both HF mirrors cite)",
    "url": "https://github.com/FreedomIntelligence/AceGPT/tree/main/eval/benchmark_eval/benchmarks/EXAMS_Arabic"
   }
  ],
  "status": "active",
  "subcategory": "Arabic high-school exam multiple-choice QA (5 subjects)",
  "summary": "Multiple-choice Arabic high-school exam questions across five subjects, run under two different harness names (arabic_exams, aexams) over the same 562-item set.",
  "tags": [
   "arabic",
   "knowledge",
   "multiple-choice",
   "multilingual-subset",
   "exam-qa"
  ],
  "task_format": "Multiple-choice exam question with four labelled options (A-D), single correct answer, in Arabic; scored either by log-likelihood ranking over the options or by exact match on a generated letter, depending on harness."
 }
}