{
 "body": "\n## What it measures\n\narabic_leaderboard_complete is a leaderboard configuration, not a single benchmark: it is an lm-evaluation-harness task group that runs 14 independently-authored Arabic evaluation datasets and combines their scores into one number. Three groups are native or partly-native Arabic content -- AlGhafa (reading comprehension, sentiment analysis and question answering from the TII LLM team, part natively authored and part translated from other English benchmarks), ACVA (58 separate yes/no datasets on Arabic culture, history and values, generated by GPT-3.5 Turbo for the AceGPT paper), and Arabic EXAMS (translated high-school exam questions) -- and ten are machine translations of well-known English benchmarks: ARC-Challenge, ARC-Easy, BoolQ, COPA, HellaSwag, MMLU, OpenBookQA, PIQA, RACE, SciQ and ToxiGen. A single average across these hides more than it reveals: a model could score well on the translated commonsense and science tasks while scoring poorly on ACVA's Arabic-culture questions, or vice versa, and the headline number would not show which.\n\n## How it is scored\n\nEach of the 14 component groups is scored on its own terms (mostly multiple-choice or yes/no accuracy, several of them by log-likelihood comparison rather than free-form generation), then combined using a size-weighted mean of two metrics, `acc` and `acc_norm`, as declared in the harness's `arabic_leaderboard_complete.yaml` (`weight_by_size: true`). Because arity and difficulty vary by component -- ACVA's items are binary, MMLU-style items are four-way, and several AlGhafa sub-tasks vary further -- no single random-guess baseline describes the whole aggregate.\n\n## Dataset and licence\n\nThere is no single dataset here: this page aggregates 14 component groups, some of which are themselves large multi-part collections (ACVA alone covers 58 per-topic datasets, AlGhafa nine sub-tasks). The harness's own README for this task documents a 15th group, `arabic_leaderboard_arabic_mmlu` -- a 40-subject, natively-sourced Arabic MMLU -- as part of this benchmark, but that group is absent from `arabic_leaderboard_complete.yaml`'s actual `task:` list, which names only 14 groups. Both files were read directly for this page, and the discrepancy is confirmed in the harness's own repository rather than assumed. Licensing is not a single answer either: each component dataset (AlGhafa, ACVA/AceGPT, EXAMS, and the ten translated-benchmark sources) carries its own terms, not summarised here.\n\n## Who publishes it\n\nThe Open Arabic LLM Leaderboard was introduced on 14 May 2024 by Ali Elfilali (2A2I), Hamza Alobeidli, Ruxandra Cojocaru and Basma El Amel Boussaha (Technology Innovation Institute, TII), and Cl\u00e9mentine Fourrier (Hugging Face), with Nathan Habib and Hakim Hacid credited in the project's citation. TII hosts the evaluation backend; Hugging Face hosts the public leaderboard space. This lm-evaluation-harness reproduction of the original task composition is a separate, independent contribution to that harness rather than the OALL team's own code.\n\n## Lineage\n\nThis page has no predecessor and one variant tracked in this repository, `arabic_leaderboard_light`, a 10%-sample version of the same 14 groups. None of the 14 aggregated component task groups (AlGhafa, ACVA, Arabic EXAMS, or the ten `arabic_mt_*` translated groups) has its own page in this repository yet, nor do the English benchmarks several of them translate that lack Arabic-specific pages here; boolq.md, piqa.md, openbookqa.md, mmlu.md, hellaswag.md, arc.md and toxigen.md, all in this repository, document the English originals several `arabic_mt_*` groups translate. Importantly, this configuration is itself a historical snapshot: the live OALL leaderboard has since moved to a \"v2\" composition built from different components (AlGhafa, ArabicMMLU, EXAMS, MadinahQA, AraTrust, ALRAGE, ArbMMLU-HT), evaluated through Hugging Face's LightEval rather than this harness reproduction. No id in this repository yet documents that v2 composition.\n\n## Saturation and contamination\n\nNo current top score specific to this exact 14-group composition was established from a source read for this page. The OALL project's own results dataset for this composition (`OALL/results`) was last modified in February 2025 and has not been updated since; the live leaderboard now scores a different v2 composition instead, so this page cannot responsibly cite the live leaderboard's current standings as if they measured the same thing. Contamination risk is high: every component test set is public with answers and has been for one to three years, and ten of the 14 groups are machine translations of long-public English benchmarks, so a model could have absorbed the English original, the Arabic translation, or both.\n\n## How to run it\n\n`lm_eval --tasks arabic_leaderboard_complete` runs all 14 component groups and reports the size-weighted aggregate; each component (for example `arabic_leaderboard_acva` or `arabic_leaderboard_arabic_mt_boolq`) can also be run individually. Results are not directly comparable to the live OALL space: that leaderboard runs on LightEval, evaluates a different v2 set of components, and scores everything with normalized log-likelihood accuracy exclusively, while this harness reproduction reports both `acc` and `acc_norm`. Treat a score computed with this task id as \"OALL's original 2024 composition, run through lm-evaluation-harness,\" not as \"the current Open Arabic LLM Leaderboard.\"\n\n## Reading the numbers\n\nA high arabic_leaderboard_complete score indicates broad competence across a wide, unevenly-weighted mix of native and machine-translated Arabic tasks -- not strength on any one skill, and not the same thing a current OALL submission measures. Because ten of the fourteen groups are translations of English benchmarks, part of what this aggregate rewards is a model's general multilingual transfer ability rather than anything distinctly Arabic; weigh the native-Arabic groups (AlGhafa, ACVA, Arabic EXAMS) more heavily if Arabic-specific competence is what you actually want to assess. Given the high contamination risk and the leaderboard's own move away from this composition, treat this score as a historical comparison point against older results computed the same way, not as a live signal of current standing.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "arabic_leaderboard_complete",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "OALL",
   "Open Arabic LLM Leaderboard",
   "OALL v1"
  ],
  "category": "composite",
  "contamination": {
   "note": "Every component test set is public with answers and has been for one to three years by the date of this research: ACVA and the AceGPT-derived components since 2023, AlGhafa since late 2023, and the ten machine-translated components since the leaderboard's 2024 launch. Ten of the 14 groups are machine translations of long-public English benchmarks (ARC, BoolQ, COPA, HellaSwag, MMLU, OpenBookQA, PIQA, RACE, SciQ, ToxiGen), so a model could plausibly have seen either the English original, the Arabic translation, or both. No held-out or refreshed portion is described for any of the 14 groups.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "ar"
   ],
   "license": "Varies by component dataset (AlGhafa, ACVA/AceGPT, EXAMS and each translated source benchmark carry their own licence); not established as a single licence for this page.",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "No single item count applies. This is an aggregation of 14 heterogeneous task groups, several of which are themselves multi-part: ACVA alone spans 58 per-topic yes/no datasets, and AlGhafa spans 9 sub-tasks. The harness's own `arabic_leaderboard_complete` README documents a 15th group, `arabic_leaderboard_arabic_mmlu` (a 40-subject, natively-sourced Arabic MMLU), as part of this benchmark -- but that group does not actually appear in `arabic_leaderboard_complete.yaml`'s own `task:` list, which names only 14 groups. That is an inconsistency inside the harness's own files, confirmed by reading both directly, not a finding about this repository's data; this page follows the 14-group list actually present in the aggregation config, and flags the README's 15th entry as documented but not aggregated. No source read for this page gives a combined item count across all 14 groups.\n",
   "splits": "test-set only aggregation across all 14 component groups; arabic_leaderboard_light (this repository) evaluates a 10% sample of each group's test set instead",
   "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/arabic_leaderboard_complete"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 4, slice A",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "arabic_leaderboard_complete",
   "opencompass": "",
   "other": "The live OALL leaderboard itself runs on Hugging Face's LightEval, not lm-evaluation-harness; this id is a separate lm-evaluation-harness reproduction of OALL's original (2024) task composition, contributed to the harness independently of the OALL team's own LightEval integration."
  },
  "id": "arabic_leaderboard_complete",
  "last_updated": "",
  "leaderboard_url": "https://huggingface.co/spaces/OALL/Open-Arabic-LLM-Leaderboard",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "arabic_leaderboard_light"
   ]
  },
  "measures": "arabic_leaderboard_complete is not one benchmark but a leaderboard configuration: an lm-evaluation-harness task group, named to match the Open Arabic LLM Leaderboard (OALL), that runs 14 separately-authored Arabic evaluation task groups and aggregates their scores into one weighted number. The 14 groups mix native-Arabic content -- AlGhafa (reading comprehension, sentiment analysis and question answering, built by the TII LLM team, part natively authored and part translated), ACVA (58 yes/no topic datasets on Arabic culture and values, generated by GPT-3.5 Turbo), and Arabic EXAMS (translated high-school exam questions) -- with ten \"arabic_mt_*\" groups that are machine translations of well-known English benchmarks: ARC-Challenge, ARC-Easy, BoolQ, COPA, HellaSwag, MMLU, OpenBookQA, PIQA, RACE, SciQ and ToxiGen. A model's headline score on this configuration says less about any single skill than about a broad, unevenly-weighted mix of Arabic knowledge, commonsense reasoning, reading comprehension and toxicity classification.\n",
  "metric": {
   "baseline_note": "No single random baseline applies: component tasks range from 2-way (BoolQ, COPA, ACVA's yes/no items) to 4-way (ARC, OpenBookQA, SciQ, MMLU) to other option counts, and the aggregation weights each group's contribution by its item count (`weight_by_size: true` in the harness config) rather than averaging baselines. No human baseline was established for the aggregate from a source read for this page.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy (acc and acc_norm, size-weighted mean across 14 task groups)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "Open Arabic LLM Leaderboard \u2014 Complete configuration",
  "page_kind": "family",
  "paper": {
   "arxiv": "",
   "title": "Introducing the Open Arabic LLM Leaderboard",
   "url": "https://huggingface.co/blog/leaderboard-arabic",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Ali Elfilali",
    "Hamza Alobeidli",
    "Cl\u00e9mentine Fourrier",
    "Basma El Amel Boussaha",
    "Ruxandra Cojocaru",
    "Nathan Habib",
    "Hakim Hacid"
   ],
   "org": "Open Arabic LLM Leaderboard (OALL) project: Technology Innovation Institute (TII) and 2A2I, with Hugging Face",
   "url": "https://huggingface.co/spaces/OALL/Open-Arabic-LLM-Leaderboard"
  },
  "released": "2024-05",
  "repo_url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/arabic_leaderboard_complete",
  "saturation": {
   "as_of": "",
   "note": "No current top score specific to this 14-group composition was established from a source read for this page. The OALL project's own `OALL/results` dataset, which holds scored runs against this exact task composition, was last modified in February 2025 and is no longer being updated: the live leaderboard has since moved to a \"v2\" composition (tracked in a separate `OALL/v2_results` dataset) built from a different set of components -- AlGhafa, ArabicMMLU, EXAMS, MadinahQA, AraTrust, ALRAGE and ArbMMLU-HT -- evaluated with Hugging Face's LightEval rather than through this lm-evaluation-harness reproduction. A v2 leaderboard score (currently topped by chat models scoring in the high 70s on that different scale) is not a like-for-like reading of this page's 14-group composition and this page does not cite it as such.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "arabic_leaderboard_complete tasks directory, lm-evaluation-harness",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/arabic_leaderboard_complete"
   },
   {
    "accessed": "2026-09-08",
    "title": "arabic_leaderboard_complete README: task groups, citations and provenance",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/arabic_leaderboard_complete/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "arabic_leaderboard_complete.yaml aggregation config (14-task list, weighting)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/arabic_leaderboard_complete/arabic_leaderboard_complete.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "Open Arabic LLM Leaderboard space (live; now runs a different, v2 task composition)",
    "url": "https://huggingface.co/spaces/OALL/Open-Arabic-LLM-Leaderboard"
   },
   {
    "accessed": "2026-09-08",
    "title": "OALL space metadata, Hugging Face API",
    "url": "https://huggingface.co/api/spaces/OALL/Open-Arabic-LLM-Leaderboard"
   },
   {
    "accessed": "2026-09-08",
    "title": "Introducing the Open Arabic LLM Leaderboard (Hugging Face blog, 2024-05-14)",
    "url": "https://huggingface.co/blog/leaderboard-arabic"
   },
   {
    "accessed": "2026-09-08",
    "title": "OALL/results dataset metadata (frozen v1 results archive, last modified 2025-02)",
    "url": "https://huggingface.co/api/datasets/OALL/results"
   },
   {
    "accessed": "2026-09-08",
    "title": "OALL/requests_v2 dataset metadata (current v2 submission queue)",
    "url": "https://huggingface.co/api/datasets/OALL/requests_v2"
   },
   {
    "accessed": "2026-09-08",
    "title": "AceGPT, Localizing Large Language Models in Arabic (source paper for the ACVA component)",
    "url": "https://arxiv.org/abs/2309.12053"
   }
  ],
  "status": "superseded",
  "subcategory": "leaderboard aggregating 14 Arabic-language task groups (native and machine-translated) into one weighted score",
  "summary": "An lm-evaluation-harness reproduction of the original Open Arabic LLM Leaderboard: 14 native and machine-translated Arabic task groups aggregated into one size-weighted accuracy score.",
  "tags": [
   "composite",
   "arabic",
   "leaderboard",
   "multiple-choice",
   "machine-translated",
   "native-sourced",
   "knowledge",
   "safety"
  ],
  "task_format": "A group-of-groups: each of the 14 component tasks is itself multiple-choice or yes/no question answering (several, such as AlGhafa and Arabic MMLU, are themselves further subdivided into many per-topic or per-subject tasks). No single prompt format applies across the whole aggregation.\n"
 }
}