{
 "body": "\n## What it measures\n\nLegalSupport asks which of two short case descriptions better supports a legal\npassage. The passage is an assertion taken from a US opinion. Each option is a\nparenthetical that lawyers wrote to explain a citation. The gold label is not a\nlawyer's extra annotation of \"correct law\"; it follows Bluebook introductory\nsignals that already rank strength of support (for example see ahead of see\nalso). When a sentence had more than two such supports, the authors sampled\ntwo. English only. Binary choice. It is a HELM targeted reasoning scenario,\nnot a [legalbench](legalbench.md) task, though Neel Guha is an author of both.\n\n## How it is scored\n\nThe default HELM run is three-shot joint multiple choice with the instruction\n\"Which statement best supports the passage?\". schema_classic.yaml headlines\nquasi_exact_match on the test split. The run spec attaches\nget_exact_match_metric_specs(). The paper appendix calls the metric accuracy\nand says the evaluation set is balanced across A and B, so chance is 50%.\nThose three names are close but not identical; a quoted number should say\nwhich column it came from. Changing the multiple-choice adapter method is a\ndocumented HELM ablation and will move the score.\n\n## Dataset and licence\n\nAppendix E.3.9 of arXiv 2211.09110 gives 13,862 train, 3,125 development and\n3,047 test items (20,034 total), with mean input length 137 GPT-2 tokens. HELM\nmaps the development file to its valid split and scores test. The scenario\nunpacks a Google Drive zip into legal_support/{train,dev,test}.jsonl. Fields\nused are context, citation_a.parenthetical, citation_b.parenthetical, and\nlabel in {a,b}. No dataset licence string appears in the scenario, the schema,\nor the appendix opened here, so license is left empty. HELM's own code is\nApache-2.0. Opinions come from CAP (case.law); CAP's terms were not opened\nfor this page.\n\n## Who publishes it\n\nStanford CRFM released LegalSupport with HELM (arXiv 2211.09110, November\n2022). Neel Guha designed and implemented the scenario, as credited in the\npaper's author contributions. The classic HELM leaderboard still lists the\nname. HELM entered maintenance mode on 2026-06-01. The scenario docstring\npoints remaining construction questions to nguha@stanford.edu and notes\nrelated then-ongoing legal-reasoning work (later LegalBench).\n\n## Lineage\n\nLegalSupport has no predecessor page in this repository. It is not a\n[legalbench](legalbench.md) subset: LegalBench is a 162-task collaboration\nfrom 2023, and the LegalBench HELM scenario file does not name this id.\n[legal_summarization](legal_summarization.md) is document summarization.\n[lsat_qa](lsat_qa.md) is LSAT analytical reasoning. schema_classic.yaml's\ntaxonomy line lists language as synthetic; the appendix describes mined real\nopinions, not invented prose.\n\n## Saturation and contamination\n\nNo current top score was read from a rendered leaderboard. The paper's 0%\nfigure for code models is a 2022 observation, not a ceiling. Contamination\nrisk is medium: labels travel with a public zip, and the source opinions are\nordinary published case law. A model that memorised Bluebook signal\nconventions could also score well without comparing the two parentheticals.\n\n## How to run it\n\nHELM: `legal_support` (optional `method=` on the multiple-choice adapter).\nDefault is joint multiple choice with three in-context examples because the\npassages are long. Do not compare a joint-scoring run with a separate\nper-choice likelihood run. lm-eval and OpenCompass names were not found.\nDownloading the Drive zip is required; this page did not verify that the live\nfile still matches the 2022 split counts.\n\n## Reading the numbers\n\nA score near 50% on the 3,047-item test split is chance on a balanced binary\nitem. A high score means the model preferred the parenthetical whose\nintroductory signal was stronger in the original opinion, not that it would\nwin a motion. It does not measure statutory interpretation, open-ended\ndrafting, or multilingual law. Pair it with [legalbench](legalbench.md) for\nbroader legal skills, and say whether the run used joint multiple choice and\nthree shots.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "legal_support",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "legal support",
   "HELM LegalSupport"
  ],
  "category": "domain",
  "contamination": {
   "note": "Items are mined from public US opinions after 1965 (CAP / case.law) and the labelled JSONL has been downloadable from the URL in the scenario since the HELM release. Gold labels are in that file. The construction uses citation signals rather than a held-out exam, so leakage is plausible for models trained on legal web text.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 3047,
   "size_note": "HELM paper appendix E.3.9 (arXiv 2211.09110) states train/dev/test splits 13,862 / 3,125 / 3,047 (20,034 total). Inputs average 137 GPT-2 tokens; outputs average 1 token. HELM main_split is test, so 3,047 is the scored headline count. The scenario downloads a zip from Google Drive (file id in legal_support_scenario.py) with train.jsonl, dev.jsonl and test.jsonl. This session did not unpack that zip; counts are from the paper.\n",
   "splits": "train 13,862 / dev (HELM valid) 3,125 / test 3,047",
   "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/legal_support_scenario.py"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-011 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-011"
  },
  "harness": {
   "bigbench": "",
   "helm": "legal_support",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Run spec legal_support,method=<adapter>; default ADAPT_MULTIPLE_CHOICE_JOINT, max_train_instances=3. schema_classic lists legal_support under reasoning and under the multiple-choice ablation group."
  },
  "id": "legal_support",
  "last_updated": "",
  "leaderboard_url": "https://crfm.stanford.edu/helm/classic/latest/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "LegalSupport is a comparative legal-entailment task introduced in HELM. Each item is a passage (a legal assertion) plus two parenthetical case descriptions. The model must pick the parenthetical that more forcefully supports the assertion. Labels come from Bluebook introductory signals (for example see versus see also) mined with the parentheticals from US state and federal opinions written after 1965, using the Caselaw Access Project. Neel Guha designed the scenario. English legal text. Two-way multiple choice, not [legalbench](legalbench.md).\n",
  "metric": {
   "baseline_note": "Two options. The HELM paper says the evaluation set is balanced in how often A or B is correct, so uniform chance is 50%. ScenarioMetadata names quasi_exact_match; get_legal_support_spec uses get_exact_match_metric_specs(). The appendix says accuracy. Report which HELM metric column you read. No human baseline is given.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "quasi_exact_match (schema); run spec also attaches exact-match metrics",
   "random_baseline": 50.0,
   "unit": "%"
  },
  "name": "LegalSupport",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2211.09110",
   "title": "Holistic Evaluation of Language Models",
   "url": "https://arxiv.org/abs/2211.09110",
   "year": 2022
  },
  "publisher": {
   "authors": [
    "Neel Guha"
   ],
   "org": "Stanford CRFM (HELM)",
   "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/legal_support_scenario.py"
  },
  "released": "2022-11",
  "repo_url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/legal_support_scenario.py",
  "saturation": {
   "as_of": "",
   "note": "The HELM paper reports 0% for the code models it tried on LegalSupport, which is a 2022-era slice, not a current top. No numeric cell was read from the JavaScript classic leaderboard. HELM maintenance mode began 2026-06-01.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM legal_support_scenario.py (binary parenthetical choice; Drive zip; metadata quasi_exact_match)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/legal_support_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "classic_run_specs.py get_legal_support_spec (3-shot MC joint, exact-match metrics)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/run_specs/classic_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "schema_classic.yaml (LegalSupport group; quasi_exact_match; taxonomy language field listed as synthetic)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/static/schema_classic.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "Holistic Evaluation of Language Models (Liang et al., 2022)",
    "url": "https://arxiv.org/abs/2211.09110"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM paper HTML appendix E.3.9 LegalSupport (splits 13862/3125/3047; CAP mining; balanced A/B)",
    "url": "https://ar5iv.labs.arxiv.org/html/2211.09110"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM README (classic leaderboard; maintenance mode 2026-06-01)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM Apache License 2.0 (harness code, not a dataset licence)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/LICENSE"
   }
  ],
  "status": "unknown",
  "subcategory": "binary reverse entailment over US case parentheticals",
  "summary": "HELM binary legal task: choose which of two case parentheticals more strongly supports a passage mined from US opinions.",
  "tags": [
   "legal",
   "entailment",
   "multiple-choice",
   "helm",
   "reasoning"
  ],
  "task_format": "Two references, one tagged correct. HELM default adapter: 3 in-context examples, instructions \"Which statement best supports the passage?\", input noun Passage, output noun Answer, method ADAPT_MULTIPLE_CHOICE_JOINT unless overridden. Binary A/B.\n"
 }
}