{
 "body": "\n## What it measures\n\nlegal_summarization is HELM's wrapper around three English legal summarization\nsets. The model reads a truncated document and writes a summary. BillSum is US\ncongressional bill text to a bill summary. MultiLexSum in this scenario is not\nthe original multi-document task: HELM copies the expert long summary into the\nprompt and scores the short summary. EurLexSum is an EU legal act (the\n`reference` field) paired with the Publications Office summary, English config\nonly, even though the source dataset has 24 languages. This is not\n[legalbench](legalbench.md), not [legal_support](legal_support.md), and not\nHELM's separate contract-summarization scenario.\n\n## How it is scored\n\nschema_classic.yaml names ROUGE-2 as the headline on the test split for each of\nthe three groups. HELM also records other ROUGE/BLEU-style summarization\nmetrics plus generative-harm scores. Defaults: sampling temperature 0.3;\nBillSum max_tokens 1,024; MultiLexSum 256 tokens with a two-sentence adapter\nhint; EurLexSum 2,048 tokens. Those caps come from the dataset papers as cited\nin classic_run_specs.py, then reduced \"for economic reasons\" on training\ndocument length. There is no official macro-average across the three datasets.\nA BillSum ROUGE-2 is not a MultiLexSum ROUGE-2.\n\n## Dataset and licence\n\nBillSum on Hugging Face (FiscalNote/billsum) lists CC0-1.0 and 18,949 / 3,269 /\n1,237 rows for train, test and California extra test. HELM's BillSum path\nloads hub name `billsum` and ignores `ca_test` in the scenario code. MultiLexSum\nis ODC-By for the dataset distribution, with case summaries and metadata under\nCC BY-NC on the card; HELM pins Hugging Face config `v20220616`. The card table\ngives 616 test rows with a short summary; HELM drops empty short fields, so\nscored n is that non-empty subset, not 908 cases. EurLexSum is CC-BY-4.0. The\nEnglish JSONL files counted here have 1,129 train, 187 validation and 188 test\nrows, matching the paper's 375 fully aligned acts split 187/188. HELM code is\nApache-2.0 and does not set the dataset licences.\n\n## Who publishes it\n\nJoel Niklaus added the HELM scenario in PR 1454, merged 2023-04-05. The\nunderlying sets are Kornilova and Eidelmann, BillSum (arXiv 1910.00523, ACL\nanthology D19-5406); Shen and colleagues, Multi-LexSum (arXiv 2206.10883); and\nAumiller, Chouhan and Gertz, EUR-Lex-Sum (arXiv 2210.13448). Stanford CRFM\nmaintains the harness. HELM's classic leaderboard URL still exists; the\nframework entered maintenance mode on 2026-06-01.\n\n## Lineage\n\nThis id is a HELM grouping, not a fourth corpus. Each dataset remains a\nstandalone summarization resource with its own paper. HELM also ships\nlegal_contract_summarization, [legalbench](legalbench.md) tasks, LexGLUE and\nLEXTREME; those are different scenarios. [arabic_legal](arabic_legal.md) is\nUAE-law Arabic QA, not summarization.\n\n## Saturation and contamination\n\nNo current HELM ROUGE-2 table was parsed from the JavaScript classic frontend.\nThe three public corpora predate the HELM wrapper, so web-trained models may\nhave seen bills, clearinghouse summaries, or EUR-Lex HTML. Treat contamination\nas at least medium. Truncation to 1,024\u20132,048 whitespace tokens means a high\nscore is not evidence the model read a 200-page case file.\n\n## How to run it\n\nHELM Classic: `billsum_legal_summarization`, `multilexsum_legal_summarization`,\nor `eurlexsum_legal_summarization`. All three attach groups `legal_summarization`\nand `summarization`. There is no `@run_spec_function(\"legal_summarization\")`.\nCompare numbers only within one dataset, one truncation, one max_tokens, and\nthe same ROUGE implementation. Do not treat long-to-short MultiLexSum as the\nNeurIPS multi-document setting.\n\n## Reading the numbers\n\nA high BillSum ROUGE-2 means n-gram overlap with US bill summaries on HELM's\ntruncated test bills. A high MultiLexSum score in HELM means the model\ncompressed an already-written long summary toward the short one. A high\nEurLexSum score is overlap with EU official summaries of English acts, 188 test\ndocuments in the files counted here. None of these scores is legal advice\nquality, citation correctness, or multilingual coverage. Read them beside\n[legalbench](legalbench.md) if the claim is about legal reasoning rather than\nsummary wording.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "legal_summarization",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "billsum_legal_summarization",
   "multilexsum_legal_summarization",
   "eurlexsum_legal_summarization"
  ],
  "category": "domain",
  "contamination": {
   "note": "All three source corpora are public Hugging Face datasets (BillSum 2019, MultiLexSum 2022, EurLexSum 2022). HELM's grouping has been in the repository since PR 1454 merged 2023-04-05. Gold summaries are in the downloadable files.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Mixed: BillSum CC0-1.0; MultiLexSum ODC-By (summaries/metadata CC BY-NC); EurLexSum CC-BY-4.0; HELM code Apache-2.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "Three corpora, scored separately. BillSum (FiscalNote/billsum, CC0-1.0): train 18,949, test 3,269, ca_test 1,237; HELM does not mention ca_test. Train items are further filtered to 200\u2013800 whitespace tokens; all splits truncate at 2,048 tokens. MultiLexSum (allenai/multi_lexsum, HELM config v20220616): Hugging Face card table lists 3,177/454/908 cases with 2,210/312/616 short summaries; HELM skips empty summary/short rows and uses summary/long as the input. EurLexSum English JSONL counted here: train 1,129, validation 187, test 188 (CC-BY-4.0). HELM train-length filters: MultiLexSum 100\u2013400 tokens, EurLexSum 400\u20131,600; doc_max_length 1,024 and 2,048 respectively. Exact HELM test n after filters was not re-run.\n",
   "splits": "per dataset train/validation/test as published; BillSum has no validation split; HELM main_split test",
   "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/legal_summarization_scenario.py"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-011 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-011"
  },
  "harness": {
   "bigbench": "",
   "helm": "billsum_legal_summarization",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Also multilexsum_legal_summarization and eurlexsum_legal_summarization in classic_run_specs.py. Group name legal_summarization. Scenario.name summarization is not a @run_spec_function. HELM also has a separate legal_contract_summarization scenario, not this id."
  },
  "id": "legal_summarization",
  "last_updated": "",
  "leaderboard_url": "https://crfm.stanford.edu/helm/classic/latest/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "legal_summarization is HELM's group over three published legal summarization corpora, not a new item set. The model receives a truncated source document and must write a summary. BillSum uses US bill text. MultiLexSum, in this harness, feeds the expert long summary and asks for the short summary, not the raw multi-document case file. EurLexSum uses the English EU legal act and its official summary. HELM currently loads only English EurLexSum. Generation, English text.\n",
  "metric": {
   "baseline_note": "schema_classic.yaml sets main_name rouge_2 and main_split test for all three run groups, with additional summarization_metrics, bias and toxicity. get_summarization_metric_specs is passed a task key matching each dataset. No official random or human baseline is defined in the scenario. There is no published average across BillSum, MultiLexSum and EurLexSum.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "rouge_2",
   "random_baseline": null,
   "unit": ""
  },
  "name": "Legal summarization (HELM)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [
    "Joel Niklaus",
    "Anastassia Kornilova",
    "Vladimir Eidelmann",
    "Zejiang Shen",
    "Dennis Aumiller"
   ],
   "org": "Stanford CRFM (HELM); datasets from FiscalNote, Allen AI / Civil Rights Litigation Clearinghouse, and Heidelberg EUR-Lex-Sum",
   "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/legal_summarization_scenario.py"
  },
  "released": "2023-04",
  "repo_url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/legal_summarization_scenario.py",
  "saturation": {
   "as_of": "",
   "note": "No numeric HELM classic leaderboard cell was read (the public page is a JavaScript app). HELM entered maintenance mode on 2026-06-01. The three datasets remain separately reported in summarization papers.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM legal_summarization_scenario.py (BillSum, MultiLexSum long\u2192short, EurLexSum English, ROUGE-2 metadata)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/legal_summarization_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "classic_run_specs.py (three run specs, length caps, temperature 0.3)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/run_specs/classic_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "schema_classic.yaml (rouge_2 main_metric for the three groups)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/static/schema_classic.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "FiscalNote/billsum card (CC0-1.0; train 18949, test 3269, ca_test 1237)",
    "url": "https://huggingface.co/datasets/FiscalNote/billsum"
   },
   {
    "accessed": "2026-09-08",
    "title": "allenai/multi_lexsum card (ODC-By / CC BY-NC summaries; split table; v20220616)",
    "url": "https://huggingface.co/datasets/allenai/multi_lexsum"
   },
   {
    "accessed": "2026-09-08",
    "title": "dennlinger/eur-lex-sum card (CC-BY-4.0; 24 EU languages; 187/188 aligned val/test)",
    "url": "https://huggingface.co/datasets/dennlinger/eur-lex-sum"
   },
   {
    "accessed": "2026-09-08",
    "title": "BillSum: A Corpus for Automatic Summarization of US Legislation",
    "url": "https://arxiv.org/abs/1910.00523"
   },
   {
    "accessed": "2026-09-08",
    "title": "Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularities",
    "url": "https://arxiv.org/abs/2206.10883"
   },
   {
    "accessed": "2026-09-08",
    "title": "EUR-Lex-Sum: A Multi- and Cross-lingual Dataset for Long-form Summarization in the Legal Domain",
    "url": "https://arxiv.org/abs/2210.13448"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM PR 1454 Add legal summarization scenarios (Joel Niklaus, merged 2023-04-05)",
    "url": "https://github.com/stanford-crfm/helm/pull/1454"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM README (maintenance mode from 2026-06-01; classic leaderboard URL)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM Apache License 2.0 (harness code)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/LICENSE"
   }
  ],
  "status": "unknown",
  "subcategory": "HELM English legal-document summarization (BillSum, MultiLexSum, EurLexSum)",
  "summary": "HELM group that scores English summaries of US bills, US civil-rights case writeups, and EU acts with ROUGE-2.",
  "tags": [
   "legal",
   "summarization",
   "helm",
   "billsum",
   "multilexsum",
   "eurlexsum",
   "rouge"
  ],
  "task_format": "Prompt pattern in the scenario docstring: \"Summarize the given document. Document: \u2026 Summary: \u2026\". Three @run_spec_function names share groups legal_summarization and summarization. Temperature default 0.3. Scenario class .name is \"summarization\", which is not a run spec.\n"
 }
}