{
 "body": "\n## What it measures\n\nEgyHellaSwag is [HellaSwag](hellaswag.md) rendered in Egyptian Arabic. The model reads an activity label and a context, then chooses one of four endings. Contexts still come from ActivityNet captions and WikiHow, not from Egyptian daily life written anew. The test is whether a model can do commonsense continuation in Masri. It is not [EgyMMLU](egymmlu.md).\n\n## How it is scored\n\nlm-eval task `egyhellaswag` is four-way multiple choice. It reports mean `acc` and length-normalized `acc_norm`. Chance is 25%. The YAML scores the 10,042-row validation split. The 10-row train split is too small to treat as a reported set. NileChat Table 1 uses zero-shot accuracy on the EGY HellaSwag column; that paper also reports 3-shot in Appendix D. Those two protocols are not interchangeable.\n\n## Dataset and licence\n\nHugging Face `UBC-NLP/EgyHellaSwag` has 10 train and 10,042 validation rows. That validation size matches English HellaSwag. There is no test split. The card language tag is `arz`. The card states MIT and links `hendrycks/test` (the MMLU repo). English HellaSwag itself is MIT via `rowanz/hellaswag`. Treat the SPDX as MIT as published, and treat that MMLU URL as a card error.\n\n## Who publishes it\n\nUBC-NLP authors Abdellah El Mekki, Houdaifa Atou, Omer Nacar, Shady Shehata, and Muhammad Abdul-Mageed introduced the set in NileChat (arXiv:2505.18383; EMNLP 2025). The Hub dataset was created 2025-05-24. EleutherAI lm-eval hosts the runnable task.\n\n## Lineage\n\nPredecessor is [hellaswag](hellaswag.md). NileChat also evaluates a Moroccan HellaSwag from Shang et al. (2025); that set has no page here. EgyHellaSwag is not a subset page of a HellaSwag family, because `hellaswag` is a standalone benchmark.\n\n## Saturation and contamination\n\nEnglish HellaSwag is saturated. This dialect copy is not shown to be at ceiling. Small-model zero-shot figures in NileChat sit in the high 20s to mid 30s. The English validation labels are old and public, so contamination risk for the underlying situations is high. Translation noise is a second failure mode: Appendix C rates sampled items on correctness and dialectness (1\u20135), not on task accuracy.\n\n## How to run it\n\n```\nlm_eval --model hf --model_args pretrained=<model> --tasks egyhellaswag\n```\n\nDo not compare `acc` with `acc_norm`, or zero-shot with 3-shot, without naming the setting. The translator named in lm-eval/HF (`google/gemma-3-27b-it`) disagrees with the NileChat paper (Command R+ following Shang et al.). Say which description you trust.\n\n## Reading the numbers\n\nA strong EgyHellaSwag score means the model ranked the translated ending that matches the English gold. It does not mean the model knows Egyptian social practice. Weak scores can be dialect, translation artifacts, or ordinary HellaSwag difficulty. Read it beside English [hellaswag](hellaswag.md) and [egymmlu](egymmlu.md), not as a replacement for either.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "egyhellaswag",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "EgyHellaswag",
   "UBC-NLP/EgyHellaSwag"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "The English HellaSwag validation labels have been public since 2019. This set is a public translation of that split. Translation quality is imperfect (NileChat Appendix C: human and LLM judges on 1-5 correctness/dialectness). No dedicated memorisation study of the Arabic text was opened.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "arz"
   ],
   "license": "MIT",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 10052,
   "size_note": "Hugging Face UBC-NLP/EgyHellaSwag: train 10 + validation 10,042 = 10,052. The validation count matches English HellaSwag validation (10,042). There is no test split (English test labels remain held out). lm-eval scores validation_split and leaves test_split null.\n",
   "splits": "train 10 / validation 10,042; no test split",
   "url": "https://huggingface.co/datasets/UBC-NLP/EgyHellaSwag"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-040 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "GPT-5.6 Luna independent review, luna-batch-040"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "egyhellaswag",
   "opencompass": "",
   "other": ""
  },
  "id": "egyhellaswag",
  "last_updated": "2025-11",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "hellaswag",
   "successors": [],
   "variants": []
  },
  "measures": "EgyHellaSwag is a machine-translated Egyptian Arabic (Masri / ISO arz) version of HellaSwag. Each item gives an activity label and a context sentence plus four endings. The model must pick the plausible continuation. It tests dialectal commonsense sentence completion, not Egyptian cultural knowledge written from scratch. Items keep original HellaSwag source_id values (ActivityNet and WikiHow).\n",
  "metric": {
   "baseline_note": "Four endings, so uniform chance is 25%. Original English HellaSwag human accuracy was 95.6%; no Egyptian-Arabic human score is published. NileChat reports translation quality on a 1-5 scale, not task accuracy.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy (acc); also acc_norm",
   "random_baseline": 25.0,
   "unit": "%"
  },
  "name": "EgyHellaSwag",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2505.18383",
   "title": "NileChat: Towards Linguistically Diverse and Culturally Aware LLMs for Local Communities",
   "url": "https://arxiv.org/abs/2505.18383",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Abdellah El Mekki",
    "Houdaifa Atou",
    "Omer Nacar",
    "Shady Shehata",
    "Muhammad Abdul-Mageed"
   ],
   "org": "UBC-NLP (University of British Columbia)",
   "url": "https://huggingface.co/datasets/UBC-NLP/EgyHellaSwag"
  },
  "released": "2025-05",
  "repo_url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/egyhellaswag",
  "saturation": {
   "as_of": "",
   "note": "NileChat Table 1 reports zero-shot EGY HellaSwag accuracy for small models (for example Qwen3-1.7B 28.44, ar-stablelm-2-chat 34.79). No current frontier cell was read. English HellaSwag is saturated; this dialect translation is not established as saturated.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "lm-eval EgyHellaSwag README (gemma-3-27b-it translation note)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/egyhellaswag/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "egyhellaswag.yaml (task name, UBC-NLP/EgyHellaSwag, acc and acc_norm)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/egyhellaswag/egyhellaswag.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "utils.process_docs (activity_label + ctx, endings, label)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/egyhellaswag/utils.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "UBC-NLP/EgyHellaSwag card (arz; train 10 / validation 10042; MIT link)",
    "url": "https://huggingface.co/datasets/UBC-NLP/EgyHellaSwag"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hub API (created 2025-05-24; lastModified 2025-11-11)",
    "url": "https://huggingface.co/api/datasets/UBC-NLP/EgyHellaSwag"
   },
   {
    "accessed": "2026-09-08",
    "title": "datasets-server split counts",
    "url": "https://datasets-server.huggingface.co/info?dataset=UBC-NLP/EgyHellaSwag"
   },
   {
    "accessed": "2026-09-08",
    "title": "NileChat (arXiv:2505.18383)",
    "url": "https://arxiv.org/abs/2505.18383"
   },
   {
    "accessed": "2026-09-08",
    "title": "NileChat HTML (Command R+ translation pipeline; Table 1 EGY HellaSwag)",
    "url": "https://arxiv.org/html/2505.18383"
   },
   {
    "accessed": "2026-09-08",
    "title": "NileChat ACL Anthology (EMNLP 2025)",
    "url": "https://aclanthology.org/2025.emnlp-main.556/"
   },
   {
    "accessed": "2026-09-08",
    "title": "Original HellaSwag paper",
    "url": "https://arxiv.org/abs/1905.07830"
   }
  ],
  "status": "active",
  "subcategory": "Egyptian Arabic commonsense sentence completion (translated HellaSwag)",
  "summary": "Egyptian Arabic four-way sentence completion translated from HellaSwag; lm-eval scores the 10,042-item validation split.",
  "tags": [
   "egyptian-arabic",
   "commonsense",
   "multiple-choice",
   "translation",
   "hellaswag"
  ],
  "task_format": "Four-way multiple choice. lm-eval task egyhellaswag concatenates activity_label and ctx as the query, uses endings as choices, and scores the integer label. Metrics: acc and acc_norm. Training split is 10 rows; scoring uses validation.\n"
 }
}