{
 "body": "\n## What it measures\n\narabic_legal tests short-answer knowledge of United Arab Emirates law in Arabic. Each of 200\nitems has a question, a gold answer, and a context passage taken from a named federal statute\nfile. Closed-book QA (`arabic_legal_qa`) sends only the question. Open-book RAG\n(`arabic_legal_rag`) puts the statute passage above the question. Both adapters ask for a brief\nModern Standard Arabic answer in the setting of UAE law, with no extra commentary. HELM's schema\ndescribes the questions as written by Arabic legal experts. The files opened for this page do\nnot name those experts.\n\nThe assigned id is the parent Scenario class `arabic_legal`. The runnable names are\n`arabic_legal_qa` and `arabic_legal_rag`. It is not [legalbench](legalbench.md), which is English\nand mostly classification. It is not the finance slice of the same repo\n([arabic_finance](arabic_finance.md)).\n\n## How it is scored\n\nAn annotator model (`openai/gpt-5.4-2026-03-05`, temperature 0) reads the question, the\nreference answer and the model output, then must emit a 0 or 1 for equivalence. The metric\n`model_judged_score` is the mean of those scores. Changing the judge changes the number. The\nparent class's `get_metadata()` instead lists `exact_match` as `main_metric`, while\nschema_arabic_enterprise.yaml lists `model_judged_score` for both QA and RAG. A published figure\nshould say which headline was used. There is no random baseline for free-form legal answers, and\nno official average of the two protocols.\n\nThe generation adapter default allows up to five in-context examples, but the scenario only\nloads the test split, so these runs are zero-shot unless another HELM path injects train items\n(not established from the files opened here). Max output tokens is 1000.\n\n## Dataset and licence\n\n`stanford-crfm/arabic-enterprise`, config `legal`, split `test`, has 200 rows. Direct count of\nlegal.csv matches datasets-server. Every row has context. Filenames point at UAE federal\ninstruments (commercial companies, maritime law, civil transactions, bankruptcy, tax, and\nothers). Both protocols use the same 200 rows; RAG is not a second dataset. The card licence is\nCC BY 4.0. The card calls the whole hub repo a proposed enterprise dataset. Created 20 April\n2026; HELM pins revision `35e114eda2e3450e0e69cf6bda9d3a2f54bf6f26`. Answers and contexts are\npublic. HELM's code licence is Apache-2.0 and does not replace the dataset licence.\n\n## Who publishes it\n\nStanford CRFM publishes the dataset and the HELM scenarios. The scenario file lives beside the\nfinance and content-generation Arabic Enterprise code. No paper or named legal-author list was\nfound on the dataset card. No public leaderboard URL was found.\n\n## Lineage\n\nThis is the legal config of Arabic Enterprise, next to [arabic_finance](arabic_finance.md) and\nan article-generation config that is not this assignment. It is not the Open Arabic LLM\nLeaderboard aggregations. Run specs are labelled EXPERIMENTAL. No successor id exists in this\nrepository.\n\n## Saturation and contamination\n\nNo top score is recorded. Two hundred short answers judged by one GPT-5.4 prompt are a small,\njudge-dependent set. Because the gold answers are public, a closed-book number after April 2026\ncan reflect memorization of this CSV rather than statute knowledge. The open-book gap (RAG minus\nQA) is the more informative contrast: it asks whether the model can use the supplied passage\nwhen it cannot recall the answer.\n\n## How to run it\n\n`pip install crfm-helm`, then `helm-run --run-entries arabic_legal_qa --suite my-suite` or\n`arabic_legal_rag`. Do not pass `arabic_legal` without a suffix; that is not a run-spec function.\nHELM has been in maintenance mode since 1 June 2026. Compare only the same protocol and the same\njudge. Exact string match against the reference, if someone reports it from `get_metadata()`, is\nnot the schema headline.\n\n## Reading the numbers\n\nA high closed-book `model_judged_score` means the configured GPT-5.4 judge thought the Arabic\noutput matched the gold short answer. That is not a lawyer's grade and not a guarantee of\ncorrect citation. A high open-book score with a low closed-book score means the model needed the\nstatute passage. Scores from another judge, or exact_match, are different evaluations. Read\n[arabic_finance](arabic_finance.md) for textbook finance items from the same hub repo, and\n[legalbench](legalbench.md) for English legal-reasoning tasks with a different skill mix.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "arabic_legal",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "arabic_legal_qa",
   "arabic_legal_rag",
   "Arabic Enterprise legal"
  ],
  "category": "domain",
  "contamination": {
   "note": "Questions, answers and contexts have been public in legal.csv since 20 April 2026 (dataset createdAt; lastModified 29 April 2026). HELM pins revision 35e114eda2e3450e0e69cf6bda9d3a2f54bf6f26. Closed-book scores can be inflated by memorizing those 200 answers. Open-book scores still leak the gold answer in the public file, but the prompt also supplies the statute passage.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "ar"
   ],
   "license": "CC-BY-4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 200,
   "size_note": "200 test rows in stanford-crfm/arabic-enterprise config legal (Hugging Face datasets-server and a direct count of legal.csv). Every row has a non-empty context. original_filename values are Arabic statute filenames (for example federal decrees on commercial companies, maritime law, civil transactions, bankruptcy). Both HELM protocols iterate the same 200 rows; RAG does not add extra items.\n",
   "splits": "single test split, 200 rows; HELM runs the same rows as qa or rag",
   "url": "https://huggingface.co/datasets/stanford-crfm/arabic-enterprise"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-004 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-004"
  },
  "harness": {
   "bigbench": "",
   "helm": "arabic_legal_qa",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "arabic_legal_rag. Parent Scenario.name arabic_legal is not a @run_spec_function."
  },
  "id": "arabic_legal",
  "last_updated": "2026-04",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "arabic_legal is HELM's legal slice of stanford-crfm/arabic-enterprise. Each item is an open-ended question in Arabic about United Arab Emirates law, paired with a short Arabic reference answer and a statute-like context passage. HELM's schema says Arabic legal experts wrote the questions. Two protocols share the same 200 rows: closed-book QA sends only the question; open-book RAG prepends the context, then a blank line, then the question. The adapter instruction in both cases tells the model to answer briefly in Modern Standard Arabic in the setting of UAE law, and to emit the answer only. This is not [legalbench](legalbench.md) (English lawyer-authored classification tasks) and not [arabic_exams](arabic_exams.md).\n",
  "metric": {
   "baseline_note": "Headline metric in schema_arabic_enterprise.yaml and ArabicLegalMetric is model_judged_score: the mean of 0/1 equivalence judgements from ArabicLegalAnnotator (openai/gpt-5.4-2026-03-05, temperature 0). ScenarioMetadata on the parent class instead names exact_match as main_metric. Those two headlines disagree; reported numbers should say which one was used. No random or human baseline is defined for this free-form task. There is no official average of QA and RAG.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100.0,
   "name": "model_judged_score",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "Arabic Legal (HELM Arabic Enterprise)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [
    "Yifan Mai (HELM scenario and dataset commits, 2026)"
   ],
   "org": "Stanford CRFM (HELM Arabic Enterprise)",
   "url": "https://huggingface.co/datasets/stanford-crfm/arabic-enterprise"
  },
  "released": "2026-04",
  "repo_url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/arabic_legal_scenario.py",
  "saturation": {
   "as_of": "",
   "note": "No public Arabic Enterprise leaderboard URL was found. The dataset card calls this a proposed dataset. Run specs are marked EXPERIMENTAL. HELM entered maintenance mode on 2026-06-01. No model card in this repository currently cites this id.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM arabic_legal_scenario.py (qa vs rag input construction; pinned HF revision; exact_match in get_metadata)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/arabic_legal_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM arabic_enterprise_run_specs.py (arabic_legal_qa / arabic_legal_rag, UAE-law instruction, LLM-judge metrics)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/run_specs/arabic_enterprise_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM schema_arabic_enterprise.yaml (closed-book vs open-book descriptions; main_name model_judged_score)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/static/schema_arabic_enterprise.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "stanford-crfm/arabic-enterprise dataset card (CC-BY-4.0; proposed dataset)",
    "url": "https://huggingface.co/datasets/stanford-crfm/arabic-enterprise"
   },
   {
    "accessed": "2026-09-08",
    "title": "datasets-server info (legal test: 200 examples)",
    "url": "https://datasets-server.huggingface.co/info?dataset=stanford-crfm/arabic-enterprise"
   },
   {
    "accessed": "2026-09-08",
    "title": "legal.csv downloaded and counted (200 rows; context on every row; UAE statute filenames)",
    "url": "https://huggingface.co/datasets/stanford-crfm/arabic-enterprise/resolve/main/legal.csv"
   },
   {
    "accessed": "2026-09-08",
    "title": "ArabicLegalAnnotator (GPT-5.4 2026-03-05, 0/1 equivalence to the reference answer)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/annotation/arabic_legal_annotator.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "ArabicLegalMetric (model_judged_score from annotator)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/metrics/arabic_legal_metric.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face dataset API (created 2026-04-20, revision 35e114eda2e3450e0e69cf6bda9d3a2f54bf6f26)",
    "url": "https://huggingface.co/api/datasets/stanford-crfm/arabic-enterprise"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM Apache License 2.0 (harness code, not the dataset licence)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM README (maintenance mode from 2026-06-01)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/README.md"
   }
  ],
  "status": "unknown",
  "subcategory": "UAE-law Arabic short-answer QA, closed-book and open-book",
  "summary": "HELM's Arabic Enterprise legal set: 200 UAE-law questions scored by an LLM judge, in closed-book and open-book (statute-in-prompt) modes.",
  "tags": [
   "legal",
   "arabic",
   "helm",
   "uae-law",
   "llm-judge"
  ],
  "task_format": "Short-answer generation in Arabic. arabic_legal_qa: question only. arabic_legal_rag: context plus question. An LLM annotator scores 1 if the output is equivalent to the reference answer, else 0.\n"
 }
}