{
 "body": "\n## What it measures\n\nAraDiCE (Arabic Dialect and Cultural Evaluation) targets two gaps the authors identify in earlier\nArabic LLM benchmarks: dialect comprehension and cultural awareness. For dialects, the authors took six\nestablished English NLU benchmarks -- [BoolQ](boolq.md), [OpenBookQA](openbookqa.md), [PIQA](piqa.md),\n[TruthfulQA](truthfulqa.md) (its MC1 multiple-choice variant), [Winogrande](winogrande.md), and\nArabicMMLU (a dialectal companion to [MMLU](mmlu.md), which has no dedicated page in this repository) --\nand machine-translated them into Egyptian and Levantine Arabic, then had humans post-edit the\ntranslations for quality; some tasks also ship Gulf-leaning MSA and English control configs. For\ncultural awareness, the authors instead wrote new multiple-choice questions testing country-specific\nknowledge across Egypt, Jordan, Lebanon, Palestine, Qatar and Syria -- content that has no English or\nMSA original, since it tests knowledge a dialect-region speaker would have rather than a translated\ncomprehension task.\n\n## How it is scored\n\nEach constituent task keeps its own native scoring: BoolQ is yes/no accuracy, PIQA/Winogrande/\nOpenBookQA/TruthfulQA-MC1 are multiple-choice accuracy, ArabicMMLU and the cultural-knowledge tasks are\nmultiple-choice accuracy over their own subject or country splits. lm-evaluation-harness exposes all 28\nof these as separate leaf tasks under one `AraDiCE` group, but that group config does not define an\naggregate metric to roll the 28 numbers into a single AraDiCE score, unlike some other multi-task\nharness groups. In practice this means a model's \"AraDiCE score\" is really a table of per-dialect,\nper-task numbers rather than one figure, and any single reported \"AraDiCE\" number should be checked for\nwhich subset it actually averages.\n\n## Dataset and licence\n\nThe paper reports roughly 45,000 post-edited samples in total across the translated tasks. In the\nharness, this is organised as 28 tasks: ArabicMMLU in two dialects (each further broken into dozens of\nsubject-level sub-tasks spanning primary through university level), five NLU benchmarks\n(BoolQ, OpenBookQA, PIQA, TruthfulQA-MC1, Winogrande) each in up to four language/dialect configs, and\nsix country-specific cultural-knowledge tasks. The Hugging Face dataset (`QCRI/AraDiCE`) is released\nunder a CC BY-NC-SA-4.0 licence, meaning non-commercial use with attribution and share-alike terms;\nanswers are public.\n\n## Who publishes it\n\nAraDiCE was published by Basel Mousi, Nadir Durrani, Fatema Ahmad, Md. Arid Hasan, Maram Hasanain,\nTameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury and Firoj Alam, hosted under the Qatar Computing\nResearch Institute's (QCRI) organisation on Hugging Face and arXiv. The paper was posted to arXiv in\nSeptember 2024 and last revised in December 2024. No dedicated public leaderboard for the benchmark was\nfound.\n\n## Lineage\n\nAraDiCE is built by translating and dialect-adapting six pre-existing benchmarks that each have their\nown page in this repository -- [BoolQ](boolq.md), [OpenBookQA](openbookqa.md), [PIQA](piqa.md),\n[TruthfulQA](truthfulqa.md), [Winogrande](winogrande.md), and the general-purpose [MMLU](mmlu.md) family\n(via its Arabic-specific ArabicMMLU variant, which has no page of its own here) -- and adding a seventh,\nwholly new component, the cultural-knowledge benchmark, which has no predecessor. It sits alongside\nother dialect- or region-specific Arabic benchmarks tracked in this repository's census queue (for\nexample `arabic_leaderboard_complete` and `arabic_leaderboard_light`, assigned to a different batch), but\nno direct successor to AraDiCE itself was found.\n\n## Saturation and contamination\n\nNo dedicated public leaderboard tracks AraDiCE scores, and no source consulted gives a recent,\nmodel-by-model top score across its 28 sub-tasks, so a saturation reading was not established. The paper\nitself reports that Arabic-specific fine-tuned models (Jais, AceGPT) outperformed general multilingual\nmodels on the dialectal tasks at publication, while noting that dialect identification, generation and\ntranslation remained challenging across the board. The dataset has been public on Hugging Face since\nSeptember 2024, roughly two years before this page was written, so contamination from broad web\npretraining is plausible, though no source consulted documents it being observed.\n\n## How to run it\n\nlm-evaluation-harness implements the full suite under the `AraDiCE` group (task id `aradice` for the\n\"overall\" rollup, though see the scoring note above about the lack of a defined aggregate), with 28\nleaf tasks covering every dialect/benchmark combination plus the six cultural tasks. No inspect_evals,\nHELM, OpenCompass or BIG-bench integration was confirmed. Because scores are reported per task and per\ndialect rather than as one number, comparing two models on \"AraDiCE\" requires checking that both reports\ncover the same subset of the 28 tasks; a report that only ran the cultural-knowledge tasks, for example,\nis not comparable to one that only ran the dialectal NLU tasks.\n\n## Reading the numbers\n\nA model that scores well across AraDiCE's dialectal tasks understands Arabic beyond Modern Standard\nArabic, the register most Arabic training data and benchmarks default to, which matters because most\nArabic speakers use a regional dialect day to day. A model that scores well on the cultural-knowledge\ncomponent knows region-specific facts that a MSA-only or translation-based evaluation would not surface\nat all. Because the two halves test genuinely different things, do not treat a single \"AraDiCE\" number\nas informative without knowing whether it reflects dialect comprehension, cultural knowledge, or an\naverage across both -- and check which of the six dialects or countries were actually covered before\ncomparing two models' results.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "aradice",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "AraDICE"
  ],
  "category": "composite",
  "contamination": {
   "note": "The dataset has been public on Hugging Face since September 2024 (about two years before this page was written), so broad web-crawl contamination is plausible, but no source consulted documents a specific contamination study or the publisher flagging observed leakage.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "ar",
    "en"
   ],
   "license": "CC-BY-NC-SA-4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 45000,
   "size_note": "~45,000 post-edited samples in total across the seven translated task types, per the paper's abstract (an approximate figure, not further broken down there). lm-evaluation-harness registers 28 leaf tasks under the `AraDiCE` group: ArabicMMLU in Egyptian and Levantine dialects (each further split into dozens of subject-level sub-tasks, e.g. high-school biology, primary-school maths, university economics); BoolQ, OpenBookQA, PIQA, TruthfulQA-MC1 and Winogrande each in Egyptian, Levantine, MSA and English configs; and 6 country-specific cultural-knowledge tasks (Egypt, Jordan, Lebanon, Palestine, Qatar, Syria).",
   "splits": "Varies by constituent task; no single unified split count is reported across all 28 sub-tasks.",
   "url": "https://huggingface.co/datasets/QCRI/AraDiCE"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 4, slice D",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "aradice",
   "opencompass": "",
   "other": ""
  },
  "id": "aradice",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "AraDiCE (Arabic Dialect and Cultural Evaluation) tests two things existing Arabic benchmarks mostly did not separate: whether a model understands specific Arabic dialects, and whether it knows region-specific cultural facts. For dialect comprehension, the authors machine-translated six established English NLU benchmarks -- BoolQ, OpenBookQA, PIQA, TruthfulQA (MC1), Winogrande, and ArabicMMLU -- into Egyptian and Levantine Arabic (and, for some tasks, Gulf-leaning Modern Standard Arabic), then had humans post-edit the machine translations for quality. For cultural awareness, they built a separate, purpose-written multiple-choice benchmark covering country-specific knowledge for Egypt, Jordan, Lebanon, Palestine, Qatar and Syria, which is not a translation of anything but new content.\n",
  "metric": {
   "baseline_note": "Random baseline varies by constituent task (roughly 50% for BoolQ's yes/no format, roughly 25% for four-option multiple choice, lower for ArabicMMLU's per-subject question sets) and is not a single number for the suite as a whole. The lm-evaluation-harness `AraDiCE` group lists all 28 sub-tasks together but, unlike some other harness groups, does not define an `aggregate_metric_list` to roll them up into one score, so running the group yields 28 separate per-task numbers.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100.0,
   "name": "accuracy (acc/acc_norm per constituent task; no single combined AraDiCE score is defined by the harness)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "AraDiCE",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2409.11404",
   "title": "AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs",
   "url": "https://arxiv.org/abs/2409.11404",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Basel Mousi",
    "Nadir Durrani",
    "Fatema Ahmad",
    "Md. Arid Hasan",
    "Maram Hasanain",
    "Tameem Kabbani",
    "Fahim Dalvi",
    "Shammur Absar Chowdhury",
    "Firoj Alam"
   ],
   "org": "Qatar Computing Research Institute (QCRI)",
   "url": "https://huggingface.co/datasets/QCRI/AraDiCE"
  },
  "released": "2024-09",
  "repo_url": "https://huggingface.co/datasets/QCRI/AraDiCE",
  "saturation": {
   "as_of": "",
   "note": "No dedicated public leaderboard for AraDiCE was found (a Hugging Face Spaces search for the name returned nothing). The paper reports that Arabic-specific models (Jais, AceGPT) outperform general multilingual models on the dialectal tasks, but a specific top-score figure for the current field was not established from the sources reviewed.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness aradice task directory (README, group yaml, all 28 sub-task configs)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/aradice"
   },
   {
    "accessed": "2026-09-08",
    "title": "AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs (arXiv abstract)",
    "url": "https://arxiv.org/abs/2409.11404"
   },
   {
    "accessed": "2026-09-08",
    "title": "QCRI/AraDiCE dataset card (licence, tags, task categories)",
    "url": "https://huggingface.co/datasets/QCRI/AraDiCE"
   },
   {
    "accessed": "2026-09-08",
    "title": "QCRI/AraDiCE dataset metadata (Hugging Face API)",
    "url": "https://huggingface.co/api/datasets/QCRI/AraDiCE"
   }
  ],
  "status": "active",
  "subcategory": "Arabic dialect and cultural-knowledge suite (translated BoolQ, OpenBookQA, PIQA, TruthfulQA-MC1, Winogrande, ArabicMMLU, plus native cultural QA)",
  "summary": "A suite that re-runs six existing English NLU benchmarks in Egyptian, Levantine and Gulf/MSA Arabic, plus a native cultural-knowledge test across six Arab countries.",
  "tags": [
   "arabic",
   "dialect",
   "cultural-knowledge",
   "composite",
   "multilingual"
  ],
  "task_format": "Mixed by constituent task -- BoolQ is yes/no reading comprehension; PIQA, Winogrande, OpenBookQA and TruthfulQA-MC1 are multiple-choice commonsense or truthfulness questions; ArabicMMLU is multi-subject multiple-choice exam questions; the cultural-knowledge benchmark is multiple-choice questions about country-specific customs, history and facts. Each is presented in a specific Arabic dialect (or English, for some tasks' original-language control configs)."
 }
}