{
 "body": "\n## What it measures\n\nThe model sees one English phrase from a legal opinion and must output\npositive, negative, or neutral. HELM loads the OSF spreadsheets that\naccompany Ratnayaka, de Silva, Perera, and Pathirana (2020). Example\nstrings cite US doctrine and courts (Bivens, Arkansas corrections, the\nEleventh Circuit). The paper\u2019s authors are at Moratuwa and Colombo; the\nHELM taxonomy\u2019s \u201cUnited States courts\u201d label matches the quoted language,\nnot the author location. This is sentence sentiment, not holding\nprediction and not [legalbench](legalbench.md).\n\n## How it is scored\n\nThe Enterprise schema headline is `quasi_exact_match` on test. The run spec\nalso records exact match and weighted F1 over the three name strings.\nGeneration uses a three-label instruction and the default generation\nadapter (the run spec does not set a shot count). Train has 576 rows; test\nhas 500, matching the paper\u2019s 500-phrase test mention. Class prior on test\nis uneven, so do not quote 33% as an official chance baseline.\n\n## Dataset and licence\n\nOSF project zwhm8 (\u201cLegal Sentiment Analysis\u201d) has no licence in the node\nAPI. HELM code is Apache-2.0. The train sheet has both Sentiment (\u22121/0/1)\nand Label (0/1/2). Those columns remap as (\u22121\u21920, 0\u21921, 1\u21922). HELM ignores\nSentiment and treats Label 0/1/2 as positive/negative/neutral. That name\norder does not match a reading of Sentiment as negative/neutral/positive.\nThe test sheet has only integer `label`. Until someone aligns those integers\nto the paper\u2019s prose names, treat HELM class names as a harness convention.\n\n## Who publishes it\n\nThe annotator paper is arXiv 2011.00318 (31 October 2020). Stanford CRFM\nadded the Enterprise scenario. A different LREC 2020 paper shares the\nanthology id that a naive title search can hit; that Flickr mental-health\npaper is not this dataset. No OSF licence and no scraped HELM cell.\n\n## Lineage\n\nNo predecessor id in this repository. Related HELM legal pages:\n[echr_judgment_classification](echr_judgment_classification.md),\n[casehold](casehold.md), [legal_support](legal_support.md). Those are\njudgment or holding tasks, not three-way sentiment.\n\n## Saturation and contamination\n\nSaturation is unknown. The xlsx files have been downloadable since 2020.\nOpinion quotes may also appear in public case corpora.\n\n## How to run it\n\nUse HELM entry `legal_opinion_sentiment_classification`, not scenario name\n`legal_opinion` alone. Downloads go to OSF. If you rebuild the task, decide\nexplicitly whether Label 0 is HELM\u2019s \u201cpositive\u201d or the train sheet\u2019s\nSentiment \u22121. Report which metric column you read.\n\n## Reading the numbers\n\nA high exact-match score means the model emitted the HELM class string that\nmatches the integer in the test sheet. It does not mean the model understood\nthe holding, the parties, or the procedural posture. Do not compare HELM\ngeneration to the paper\u2019s 57% BERT accuracy without mapping labels. Read\nthis beside a legal NLU suite such as [lex_glue](lex_glue.md) if you need\nbroader coverage.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "legal_opinion_sentiment_classification",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "legal_opinion"
  ],
  "category": "domain",
  "contamination": {
   "note": "OSF spreadsheets have been public since 2020-09/10. Gold integer labels are in the xlsx files. Phrases look like US opinion excerpts and may also occur in case-law crawls.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 500,
   "size_note": "OSF file Testing_Set_Legal_Sentences.xlsx has 500 data rows (header plus 500). Legal_Sentences_For_Training_With_BERT_With_Label.xlsx has 576 data rows. HELM downloads those two files from osf.io/hfn62 and osf.io/q4adh (project zwhm8, \"Legal Sentiment Analysis\", created 2020-09-13). Train Label 0/1/2 counts 282/122/172 pair exactly with Sentiment \u22121/0/1. HELM maps Label index onto [positive, negative, neutral], which does not match the usual \u22121=negative, 0=neutral, 1=positive reading of the Sentiment column. Test has only integer label, no Sentiment name.\n",
   "splits": "HELM train 576 / test 500; no validation split in the scenario",
   "url": "https://osf.io/zwhm8/"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-053 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-053"
  },
  "harness": {
   "bigbench": "",
   "helm": "legal_opinion_sentiment_classification",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Scenario.name is legal_opinion; the runnable HELM entry is legal_opinion_sentiment_classification."
  },
  "id": "legal_opinion_sentiment_classification",
  "last_updated": "",
  "leaderboard_url": "https://crfm.stanford.edu/helm/enterprise/latest/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "legal_opinion_sentiment_classification is HELM Enterprise's wrap of the phrase-level sentiment data released with Ratnayaka et al. (arXiv 2011.00318). The model reads one English sentence or fragment from a legal opinion and must answer positive, negative, or neutral. HELM does not score party-specific sentiment or the paper's BERT word-list pipeline. Text only.\n",
  "metric": {
   "baseline_note": "schema_enterprise.yaml main_name is quasi_exact_match, main_split test. The run spec adds exact-match metrics and a weighted classification metric over labels positive, neutral, negative. Test labels are not uniform (211 / 168 / 121 of 0 / 1 / 2), so chance is not 33.3% unless you assume a balanced prior. Paper reports 57% BERT accuracy on 500 phrases after two epochs of their own training, not HELM generation.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "quasi_exact_match (schema); run spec also attaches weighted classification F1",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "Legal Opinion Sentiment Classification (HELM)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2011.00318",
   "title": "Effective Approach to Develop a Sentiment Annotator For Legal Domain in a Low Resource Setting",
   "url": "https://arxiv.org/abs/2011.00318",
   "year": 2020
  },
  "publisher": {
   "authors": [
    "Gathika Ratnayaka",
    "Nisansa de Silva",
    "Amal Shehan Perera",
    "Ramesh Pathirana"
   ],
   "org": "University of Moratuwa and University of Colombo (dataset); Stanford CRFM (HELM Enterprise scenario)",
   "url": "https://osf.io/zwhm8/"
  },
  "released": "2020-10",
  "repo_url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/legal_opinion_sentiment_classification_scenario.py",
  "saturation": {
   "as_of": "",
   "note": "No numeric HELM Enterprise cell was read. HELM entered maintenance mode on 2026-06-01. The paper's 57% BERT figure is a different protocol.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM scenario (OSF URLs, class name legal_opinion, label index mapping)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/legal_opinion_sentiment_classification_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "enterprise_run_specs.py (run spec name, instructions, exact-match plus weighted F1)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/run_specs/enterprise_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "schema_enterprise.yaml (quasi_exact_match; US legal opinion taxonomy)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/static/schema_enterprise.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "OSF project Legal Sentiment Analysis (no licence field)",
    "url": "https://api.osf.io/v2/nodes/zwhm8/"
   },
   {
    "accessed": "2026-09-08",
    "title": "OSF train xlsx metadata (576-row file, 2020-10-25)",
    "url": "https://api.osf.io/v2/files/hfn62/"
   },
   {
    "accessed": "2026-09-08",
    "title": "OSF test xlsx metadata (500-row file, 2020-09-21)",
    "url": "https://api.osf.io/v2/files/q4adh/"
   },
   {
    "accessed": "2026-09-08",
    "title": "Train spreadsheet (Phrase, Sentiment, Label)",
    "url": "https://osf.io/download/hfn62/"
   },
   {
    "accessed": "2026-09-08",
    "title": "Test spreadsheet (sentence, label)",
    "url": "https://osf.io/download/q4adh/"
   },
   {
    "accessed": "2026-09-08",
    "title": "arXiv abs 2011.00318 (submitted 2020-10-31)",
    "url": "https://arxiv.org/abs/2011.00318"
   },
   {
    "accessed": "2026-09-08",
    "title": "Paper HTML (500-phrase test mentioned; Sri Lankan affiliations)",
    "url": "https://ar5iv.labs.arxiv.org/html/2011.00318"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM README (maintenance mode 2026-06-01)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM Apache License 2.0",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/LICENSE"
   }
  ],
  "status": "unknown",
  "subcategory": "three-way sentiment on English legal-opinion phrases (HELM Enterprise)",
  "summary": "HELM three-class task: label a legal-opinion phrase positive, negative, or neutral using Ratnayaka et al. OSF spreadsheets.",
  "tags": [
   "legal",
   "sentiment",
   "classification",
   "helm"
  ],
  "task_format": "Run spec legal_opinion_sentiment_classification. Scenario class name is legal_opinion. Instructions \"Classify the sentences into one of the 3 sentiment categories. Possible labels: positive, neutral, negative.\" Generation adapter, output noun Label. Train xlsx columns Phrase/Label; test xlsx columns sentence/label.\n"
 }
}