{
 "body": "\n## What it measures\n\nThe model is given a short English passage from a unilateral consumer contract\nand must write a plain-English summary. Sources are TL;DRLegal and TOS;DR,\nscraped for Manor and Li (NLLP 2019). Typical documents are terms of service,\nprivacy policies, and a few software licenses, not statutes or opinions.\nCommunity members wrote the gold summaries; the paper says neither site\nrequires legal credentials. HELM\u2019s schema line that annotators are \u201clawyers\u201d\ndoes not match that paper. This is not the HELM [legal_summarization](legal_summarization.md)\ngroup.\n\n## How it is scored\n\nschema_enterprise.yaml names ROUGE-L on the test split as the headline.\nThe run spec also records ROUGE-1 and ROUGE-2. Generation uses the instruction\n\u201cSummarize the legal document in plain English,\u201d a 100-token cap, and a blank\nline as a stop. HELM builds train and test by sampling 20% of `all_v1.json`\nwith `random_state=0`. There is no published human ROUGE on that split. The\n2019 paper\u2019s point is that extractive baselines fail because the summaries\nrewrite and compress.\n\n## Dataset and licence\n\n`all_v1.json` holds 446 uid / original_text / reference_summary records.\nThe upstream README calls this the quantitative-analysis set after dropping\nsummaries longer than the source. The paper describes 84 TL;DRLegal sets from\nnine company documents and hundreds of TOS;DR sets later filtered by the\nfirst author. No LICENSE file exists in that GitHub repo. The paper page\nmarks TOS;DR material CC BY-SA 3.0; a combined SPDX id is not established.\nGold summaries are public. HELM code is Apache-2.0.\n\n## Who publishes it\n\nLaura Manor and Junyi Jessy Li (UT Austin Linguistics) introduced the task\nat the 2019 NAACL NLLP workshop (7 June, Minneapolis). Stanford CRFM wrapped\nit as a HELM Enterprise scenario. The Enterprise leaderboard URL is the\nHELM page; this session did not scrape a numeric cell from that JavaScript\napp.\n\n## Lineage\n\nRelated HELM legal pages in this repository:\n[legal_summarization](legal_summarization.md), [legal_support](legal_support.md),\n[legalbench](legalbench.md), and [casehold](casehold.md). Those use different\ncorpora. No successor contract-summarization id is recorded here.\n\n## Saturation and contamination\n\nSaturation is unknown without a dated HELM cell. The JSON and the two\ncommunity sites have been public since 2019, so leakage is plausible.\nHELM moved to maintenance mode on 2026-06-01.\n\n## How to run it\n\nInstall crfm-helm and run the entry `legal_contract_summarization`. Do not\nuse a `legal_summarization` BillSum spec. ROUGE will move if the 100-token\ncap, stop sequence, or 20% split seed changes. Compare only Enterprise\nschema ROUGE-L unless you re-implement the paper\u2019s own analysis.\n\n## Reading the numbers\n\nA high ROUGE-L means the model\u2019s short rewrite overlapped the community\nsummary on these snippets. It does not mean the model can summarise a full\nmaster service agreement, or that the summary is legally complete. Many\nTOS;DR lines are templatic across companies. Read the number beside a\nfaithfulness check and beside [legal_summarization](legal_summarization.md)\nif you care about bills or case writeups.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "legal_contract_summarization",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Plain English Summarization of Contracts"
  ],
  "category": "domain",
  "contamination": {
   "note": "all_v1.json has been public on GitHub since the 2019 paper. Gold summaries are in that file. Snippets are short TOS/license clauses that also appear on the live TL;DRLegal and TOS;DR sites.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 446,
   "size_note": "lauramanor/legal_summarization all_v1.json has 446 records after dropping rows missing original_text, reference_summary, or uid. HELM samples 20% as train with pandas sample(frac=0.2, random_state=0) and uses the rest as test; exact test n was not re-run here without pandas. Upstream README: 84 TL;DRLegal sets and a TOS;DR annotated file; all_v1 is the combined quantitative-analysis set excluding summaries longer than the source. 59 distinct doc titles in the JSON (Privacy Policy 111, Terms of Service 81, and others).\n",
   "splits": "HELM random 20% train / remainder test from all_v1.json; no published validation split",
   "url": "https://github.com/lauramanor/legal_summarization"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-053 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-053"
  },
  "harness": {
   "bigbench": "",
   "helm": "legal_contract_summarization",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "HELM Enterprise run spec and group legal_contract_summarization. Not the classic legal_summarization group."
  },
  "id": "legal_contract_summarization",
  "last_updated": "",
  "leaderboard_url": "https://crfm.stanford.edu/helm/enterprise/latest/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "legal_contract_summarization is HELM's wrap of Manor and Li (NLLP 2019). The model reads a cleaned snippet from a terms-of-service or software license and must write a short plain-English summary. Gold text comes from TL;DRLegal and TOS;DR community pages, not from a court. English. This is not [legal_summarization](legal_summarization.md) (BillSum / MultiLexSum / EurLexSum).\n",
  "metric": {
   "baseline_note": "schema_enterprise.yaml main_name rouge_l, main_split test. The run spec also attaches rouge_1 and rouge_2. The 2019 paper reports that unsupervised extractive methods fail because summaries are abstractive; it does not publish a human ROUGE for this HELM split.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "rouge_l",
   "random_baseline": null,
   "unit": ""
  },
  "name": "Legal Contract Summarization (HELM)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "Plain English Summarization of Contracts",
   "url": "https://aclanthology.org/W19-2201/",
   "year": 2019
  },
  "publisher": {
   "authors": [
    "Laura Manor",
    "Junyi Jessy Li"
   ],
   "org": "University of Texas at Austin (dataset); Stanford CRFM (HELM Enterprise scenario)",
   "url": "https://github.com/lauramanor/legal_summarization"
  },
  "released": "2019-06",
  "repo_url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/legal_contract_summarization_scenario.py",
  "saturation": {
   "as_of": "",
   "note": "No numeric HELM Enterprise cell was read (the public page is a JavaScript app). HELM entered maintenance mode on 2026-06-01.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM legal_contract_summarization_scenario.py (all_v1.json, 20% train, rouge_l metadata)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/legal_contract_summarization_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "enterprise_run_specs.py (run spec, max_tokens 100, rouge_1/2/l)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/run_specs/enterprise_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "schema_enterprise.yaml (main_metric rouge_l, legal_scenarios group)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/static/schema_enterprise.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "all_v1.json (446 records)",
    "url": "https://raw.githubusercontent.com/lauramanor/legal_summarization/master/all_v1.json"
   },
   {
    "accessed": "2026-09-08",
    "title": "Upstream README (446 combined sets; 84 TL;DRLegal; no SPDX file)",
    "url": "https://raw.githubusercontent.com/lauramanor/legal_summarization/master/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "NLLP 2019 anthology page (Manor and Li, June 2019)",
    "url": "https://aclanthology.org/W19-2201/"
   },
   {
    "accessed": "2026-09-08",
    "title": "PDF (community summaries; TOS;DR CC BY-SA 3.0 mention on the data page)",
    "url": "https://aclanthology.org/W19-2201.pdf"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM README (maintenance mode from 2026-06-01; Enterprise leaderboard)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM Apache License 2.0 (harness code)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/LICENSE"
   }
  ],
  "status": "unknown",
  "subcategory": "HELM Enterprise plain-English summarization of TOS and license snippets",
  "summary": "HELM generation task: rewrite a short unilateral-contract snippet in plain English and score ROUGE-L against community summaries.",
  "tags": [
   "legal",
   "summarization",
   "helm",
   "contracts",
   "rouge"
  ],
  "task_format": "HELM @run_spec_function legal_contract_summarization in enterprise_run_specs.py. Instructions \"Summarize the legal document in plain English.\" Input noun Document, output noun Summary, max_tokens 100, stop at blank line. Scenario.name matches the run spec.\n"
 }
}