{
 "body": "\n## What it measures\n\narabic_finance tests whether a model can answer short finance questions in Modern Standard\nArabic (or, optionally, in English). HELM's schema says the questions come from English finance\ntextbooks and were machine-translated into Arabic. The Hugging Face dataset stores both\nlanguages on every row. A run is one of three formats, not a mix: 23 three-option multiple-choice\nitems, 57 true/false items, and 219 calculation items that need a number. The skill mix is\ntextbook recall, a binary truth judgement, and arithmetic over a finance formula. It is not\nopen-book filing QA. That is [financebench](financebench.md). It is not Arabic high-school exams\n([arabic_exams](arabic_exams.md)).\n\nThe assigned id is the parent Scenario class `arabic_finance`. HELM does not register a run spec\nunder that exact name. The runnable names are `arabic_finance_mcq`, `arabic_finance_bool` and\n`arabic_finance_calculation`.\n\n## How it is scored\n\nMCQ uses joint multiple-choice generation. The CSV labels every choice A/B/C, including on\nArabic rows. The Arabic adapter remaps those option prefixes to \u0623, \u0628, \u062c and the model must\nemit one of those letters. In English it emits A, B or C. Headline metric is exact_match.\nThree options give a 1/3 chance rate if the labels are balanced.\n\nBool asks whether a passage is true. Arabic instructions require \u0646\u0639\u0645 or \u0644\u0627; English requires\nYes or No. The run spec attaches exact-match style metrics; schema_arabic_enterprise.yaml names\n`quasi_exact_match` as the bool headline. Those two names are not the same; a reported bool\nscore should say which one was used.\n\nCalculation asks for reasoning and a final numeric answer wrapped in `\\boxed{}`. An annotator\nmodel (`openai/gpt-5.4-2026-03-05`, temperature 0) judges whether the output is mathematically\nequivalent to the reference and writes a 0 or 1. `calculation_accuracy` is the mean of those\nscores. Changing the judge model changes the number. There is no official combined score across\nthe three formats.\n\nThe generation adapter defaults to up to five in-context examples, but the scenario only emits\na test split, so the runs are zero-shot unless some other HELM path injects train items (not\nestablished from the files opened here).\n\n## Dataset and licence\n\n`stanford-crfm/arabic-enterprise`, config `finance`, split `test`, has 299 rows. Direct count of\nfinance.csv matches the datasets-server figure. Task field: mcq 23, bool 57, calcu 219. The card\nlicence is CC BY 4.0. The card text is one sentence: a proposed dataset for enterprise LLM use\ncases in Arabic. Created 20 April 2026; HELM pins revision\n`35e114eda2e3450e0e69cf6bda9d3a2f54bf6f26`. Answers are public in both languages. HELM's code\nlicence is Apache-2.0 and does not replace the dataset licence.\n\n## Who publishes it\n\nStanford CRFM publishes the dataset and the HELM scenarios. Git history on the scenario file\nnames Yifan Mai on the 2026-03 and 2026-04 commits, including the switch to the stanford-crfm\nHugging Face org (PR 4236, 29 April 2026). No paper, author list, or public leaderboard for\nthis slice was found. The dataset card does not name textbook sources beyond HELM's schema\nsentence.\n\n## Lineage\n\nThis is one of three Arabic Enterprise configs (finance, legal, content_generation). The legal\nslice is [arabic_legal](arabic_legal.md). It is not LegalBench, FinanceBench, or the Open Arabic\nLLM Leaderboard aggregations ([arabic_leaderboard_complete](arabic_leaderboard_complete.md)).\nRun specs are labelled EXPERIMENTAL and may change. No successor id exists in this repository.\n\n## Saturation and contamination\n\nNo top score is recorded. Twenty-three MCQ items and fifty-seven bool items are small enough\nthat a single-run percentage will bounce. The 219 calculation items are the only slice large\nenough to look like a benchmark, and they depend on a specific GPT-5.4 judge. The full CSV has\nbeen public since April 2026, so models trained after that date can have seen the answers.\nTextbook stems may have been public in English earlier.\n\n## How to run it\n\n`pip install crfm-helm`, then for example\n`helm-run --run-entries arabic_finance_mcq --suite my-suite` (Arabic default),\n`arabic_finance_bool:lang=en`, or `arabic_finance_calculation`. Do not pass `arabic_finance`\nwithout a suffix; that is not a run-spec function. HELM has been in maintenance mode since\n1 June 2026. Compare only the same format, language, and metric name. Calculation scores from\na different judge are a different evaluation.\n\n## Reading the numbers\n\nA high MCQ exact_match on 23 items is a noisy three-way quiz, not a finance-professional exam.\nA high bool score is yes/no accuracy on 57 statements. A high calculation_accuracy means the\nconfigured GPT-5.4 judge accepted the boxed number as equivalent 219 times at most; it is not\na human grader and not exact string match. Never average the three formats unless you say you\ninvented that average. Read [arabic_legal](arabic_legal.md) for UAE-law questions from the same\ndataset repo, and an English open-book filing benchmark such as [financebench](financebench.md)\nif you need a different finance skill.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "arabic_finance",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "arabic_finance_mcq",
   "arabic_finance_bool",
   "arabic_finance_calculation",
   "Arabic Enterprise finance"
  ],
  "category": "domain",
  "contamination": {
   "note": "The full test CSV, including English and Arabic answers, has been public on Hugging Face since 20 April 2026 (dataset createdAt; lastModified 29 April 2026). HELM pins revision 35e114eda2e3450e0e69cf6bda9d3a2f54bf6f26. Schema text says the questions come from English finance textbooks via machine translation, so some English stems may exist in earlier web crawls even if this packaging is new.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "ar",
    "en"
   ],
   "license": "CC-BY-4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 299,
   "size_note": "299 test rows in stanford-crfm/arabic-enterprise config finance (Hugging Face datasets-server and a direct count of finance.csv). By task field: mcq 23, bool 57, calcu 219. cap_group values in the CSV are Multi 50, NM 50, SP 50, TR 50, TU 50, FF 49; HELM's schema does not define those codes, so they are not interpreted here. One test split only. Default language is Arabic; English is available via lang=en.\n",
   "splits": "single test split, 299 rows; HELM filters by task=mcq|bool|calcu",
   "url": "https://huggingface.co/datasets/stanford-crfm/arabic-enterprise"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-004 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-004"
  },
  "harness": {
   "bigbench": "",
   "helm": "arabic_finance_mcq",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "arabic_finance_bool; arabic_finance_calculation. Parent Scenario.name arabic_finance is not a @run_spec_function. Default lang=ar; English via lang=en (run name gains :lang=en)."
  },
  "id": "arabic_finance",
  "last_updated": "2026-04",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "arabic_finance is HELM's finance slice of the stanford-crfm/arabic-enterprise dataset. Each item is a short finance question drawn, per HELM's schema, from English-language finance textbooks and machine-translated into Arabic. The same 299 rows are stored with English and Arabic question, choice and answer fields. HELM splits them into three task formats: 23 three-option multiple-choice questions (task=mcq), 57 yes/no verifications (task=bool), and 219 numeric calculation problems (task=calcu). A run is one format and one language (default Arabic). It is text-only professional-finance QA, not [financebench](financebench.md) (English SEC-filing QA) and not [financeiq](financeiq.md).\n",
  "metric": {
   "baseline_note": "MCQ has three options on every item (the scenario asserts len(references)==3), so chance is 1/3 if options are balanced. Bool is two-way (Yes/No or \u0646\u0639\u0645/\u0644\u0627). Calculation is free numeric; no chance rate. There is no single combined arabic_finance score in schema_arabic_enterprise.yaml: each format is its own run group with its own main_name. Bool's run spec attaches get_exact_match_metric_specs() (exact_match plus quasi_exact_match); the schema headline for bool is quasi_exact_match. Calculation accuracy is a 0/1 LLM-judge score averaged over items.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100.0,
   "name": "exact_match (MCQ); quasi_exact_match (bool, schema headline); calculation_accuracy (calculation)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "Arabic Finance (HELM Arabic Enterprise)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [
    "Yifan Mai (HELM scenario and dataset commits, 2026)"
   ],
   "org": "Stanford CRFM (HELM Arabic Enterprise)",
   "url": "https://huggingface.co/datasets/stanford-crfm/arabic-enterprise"
  },
  "released": "2026-04",
  "repo_url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/arabic_finance_scenario.py",
  "saturation": {
   "as_of": "",
   "note": "No public Arabic Enterprise leaderboard URL was found. The dataset card calls this a proposed dataset. Run specs are marked EXPERIMENTAL. HELM entered maintenance mode on 2026-06-01. No model card in this repository currently cites this id.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM arabic_finance_scenario.py (parent class and bool/mcq/calcu subclasses; pinned HF revision)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/arabic_finance_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM arabic_enterprise_run_specs.py (run spec names, adapters, metrics; EXPERIMENTAL banner)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/run_specs/arabic_enterprise_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM schema_arabic_enterprise.yaml (headlines, textbook-MT description, main metrics)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/static/schema_arabic_enterprise.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "stanford-crfm/arabic-enterprise dataset card (CC-BY-4.0; proposed dataset; finance/legal/content_generation)",
    "url": "https://huggingface.co/datasets/stanford-crfm/arabic-enterprise"
   },
   {
    "accessed": "2026-09-08",
    "title": "datasets-server info (finance test: 299 examples)",
    "url": "https://datasets-server.huggingface.co/info?dataset=stanford-crfm/arabic-enterprise"
   },
   {
    "accessed": "2026-09-08",
    "title": "finance.csv downloaded and counted (299 rows: mcq 23, bool 57, calcu 219)",
    "url": "https://huggingface.co/datasets/stanford-crfm/arabic-enterprise/resolve/main/finance.csv"
   },
   {
    "accessed": "2026-09-08",
    "title": "ArabicFinanceCalculationAnnotator (GPT-5.4 2026-03-05 judge, 0/1 mathematical equivalence)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/annotation/arabic_finance_calculation_annotator.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "ArabicFinanceCalculationMetric (calculation_accuracy from annotator score)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/metrics/arabic_finance_calculation_metric.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face dataset API (created 2026-04-20, revision 35e114eda2e3450e0e69cf6bda9d3a2f54bf6f26)",
    "url": "https://huggingface.co/api/datasets/stanford-crfm/arabic-enterprise"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM Apache License 2.0 (harness code, not the dataset licence)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM README (maintenance mode from 2026-06-01)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/README.md"
   }
  ],
  "status": "unknown",
  "subcategory": "Arabic (and English) finance textbook QA: 3-way MCQ, yes/no, numeric calculation",
  "summary": "HELM's Arabic Enterprise finance set: 299 textbook-derived items in three formats (3-way MCQ, yes/no, numeric calculation), in Arabic or English.",
  "tags": [
   "finance",
   "arabic",
   "helm",
   "multiple-choice",
   "calculation"
  ],
  "task_format": "Three HELM run specs share one CSV. MCQ: pick one of three labelled choices (CSV uses A/B/C; the Arabic adapter remaps prefixes to \u0623/\u0628/\u062c). Bool: answer \u0646\u0639\u0645/\u0644\u0627 or Yes/No. Calculation: write reasoning, then a numeric answer inside \\\\boxed{}; an LLM annotator judges mathematical equivalence.\n"
 }
}