{
 "body": "\n## What it measures\n\nLexGLUE is an English legal NLU suite in the [GLUE](glue.md) style. A model\nreads ECtHR facts, a US Supreme Court opinion, an EU act, an SEC clause, a\nconsumer ToS sentence, or a CaseHOLD prompt, and must emit the dataset\u2019s\nlabels. ECtHR A/B and EUR-LEX and UNFAIR-ToS are multi-label. SCOTUS and\nLEDGAR are single-label. CaseHOLD is five-way QA. HELM turns each subset\ninto a generation prompt with a written instruction block. This page is the\nsuite, not a new item set.\n\n## How it is scored\n\nThe ACL paper fine-tunes encoders and reports micro-F1 and macro-F1 per\ntask, then three means across tasks. Legal-BERT is the published leader on\nthat table (79.8 / 72.0 arithmetic \u03bc-F1 / m-F1). HELM classic instead\nheadlines `classification_macro_f1` on test for generation runs\n`lex_glue:subset=...`. Shot caps are small on long documents (one shot for\nECtHR and SCOTUS; five for the shorter sets). Multi-label answers are\ncomma-separated. A HELM score is not an encoder fine-tune score. Chalkidis\n(2023) separately audited ChatGPT at 47.6% average micro-F1 zero-shot.\n\n## Dataset and licence\n\nThe Hub card `coastalcph/lex_glue` (licence CC-BY-4.0) is the current\ndataset id; HELM still calls `load_dataset(\"lex_glue\")`. Test counts on the\ncard sum to 23,607 if CaseHOLD\u2019s 3,600-row test file is used. Table 1 of\nthe paper lists CaseHOLD 45,000 / 3,900 / 3,900. SCOTUS class cardinality\nis 14 in the paper and in HELM\u2019s instruction list, but the Hub\n`class_label` names 13 codes. Labels travel with the downloads.\n\n## Who publishes it\n\nIlias Chalkidis and co-authors introduced LexGLUE on arXiv on 3 October\n2021 and at ACL 2022 (May, Dublin). Code and the encoder leaderboard live\nin coastalcph/lex-glue. Stanford CRFM added the HELM classic scenario.\nHELM itself entered maintenance mode on 2026-06-01.\n\n## Lineage\n\nPredecessor: [glue](glue.md) as the multi-task template, not as shared\nitems. [casehold](casehold.md) is the CaseHOLD dataset with its own HELM\nmultiple-choice spec. [echr_judgment_classification](echr_judgment_classification.md)\nis a binary ECHR violation task; HELM comments there that it is not\nLexGLUE ECtHR-A. [lextreme](lextreme.md) is the later multilingual legal\nsuite. [legalbench](legalbench.md) is a 162-task LLM reasoning suite, not\nthis encoder benchmark.\n\n## Saturation and contamination\n\nEncoder UNFAIR-ToS micro-F1 is already high. Other tasks, and zero-shot\ngeneration, still have headroom on the published numbers. All sources are\nold public legal corpora, so contamination risk is high.\n\n## How to run it\n\nPaper code: Hugging Face Transformers scripts in coastalcph/lex-glue.\nHELM: `lex_glue:subset=ledgar` (or another config). Wrong-answer lists are\ncapped at 30 after a seeded shuffle, which can drop EUR-LEX classes.\nMax tokens are 5\u201320 depending on subset. Do not average HELM subset scores\nunless you state that average; the paper\u2019s official means use encoder F1.\n\n## Reading the numbers\n\nA strong encoder mean means the model fits these seven legal classification\nheads. A strong HELM generation score means it can emit the same labels\nfrom a prompt. Neither is legal advice quality, retrieval, or multilingual\nlaw. Check which CaseHOLD test n you used (3,600 vs 3,900). Read\n[lextreme](lextreme.md) if you need non-English legal NLU, and\n[legalbench](legalbench.md) if you need IRAC-style LLM tasks.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "lex_glue",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "LexGLUE",
   "Legal GLUE"
  ],
  "category": "domain",
  "contamination": {
   "note": "All seven source sets were public before LexGLUE. The Hub snapshot has been downloadable since 2022-03-02 (lastModified 2024-01-04). HELM loads labels with the text.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC-BY-4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 23607,
   "size_note": "Sum of Hugging Face coastalcph/lex_glue test splits: ECtHR A 1,000; ECtHR B 1,000; SCOTUS 1,400; EUR-LEX 5,000; LEDGAR 10,000; UNFAIR-ToS 1,607; CaseHOLD 3,600 (23,607). Train/val on the same card: 9,000/1,000; 9,000/1,000; 5,000/1,400; 55,000/5,000; 60,000/10,000; 5,532/2,275; 45,000/3,900. The ACL paper Table 1 lists CaseHOLD test 3,900; the Hub test file has 3,600. SCOTUS in the paper is 14 issue areas; the Hub class_label lists 13 names while HELM instructions list 14.\n",
   "splits": "per config train/validation/test; HELM scores test",
   "url": "https://huggingface.co/datasets/coastalcph/lex_glue"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-053 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-053"
  },
  "harness": {
   "bigbench": "",
   "helm": "lex_glue:subset=<task>",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Group lex_glue. HELM load_dataset('lex_glue'); current Hub id coastalcph/lex_glue. CaseHOLD also has a standalone HELM casehold spec."
  },
  "id": "lex_glue",
  "last_updated": "2024-01",
  "leaderboard_url": "https://github.com/coastalcph/lex-glue",
  "lineage": {
   "family": "",
   "predecessor": "glue",
   "successors": [],
   "variants": [
    "casehold",
    "echr_judgment_classification"
   ]
  },
  "measures": "LexGLUE packs seven English legal datasets behind one evaluation recipe. ECtHR A predicts violated Convention articles from facts. ECtHR B predicts articles the court considered. SCOTUS maps an opinion to a Supreme Court Database issue area. EUR-LEX assigns EuroVoc labels to an EU act. LEDGAR classifies an SEC contract provision into one of 100 topics. UNFAIR-ToS tags unfair term types in a consumer ToS sentence. CaseHOLD is five-way holding identification. HELM prompts these as generation, not encoder fine-tuning.\n",
  "metric": {
   "baseline_note": "schema_classic.yaml main_name classification_macro_f1, main_split test. The original suite reports micro-F1 and macro-F1 per task, then arithmetic/harmonic/geometric means. Legal-BERT leads the published encoder table at 79.8 / 72.0 arithmetic \u03bc-F1 / m-F1. Chalkidis (2023) reports gpt-3.5-turbo zero-shot 47.6% average micro-F1. No single random baseline fits 2-way through 100-way and multi-label tasks.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "classification_macro_f1 (HELM schema); papers also report micro-F1",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "LexGLUE (Legal General Language Understanding Evaluation)",
  "page_kind": "family",
  "paper": {
   "arxiv": "2110.00976",
   "title": "LexGLUE: A Benchmark Dataset for Legal Language Understanding in English",
   "url": "https://aclanthology.org/2022.acl-long.297/",
   "year": 2022
  },
  "publisher": {
   "authors": [
    "Ilias Chalkidis",
    "Abhik Jana",
    "Dirk Hartung",
    "Michael Bommarito",
    "Ion Androutsopoulos",
    "Daniel Martin Katz",
    "Nikolaos Aletras"
   ],
   "org": "University of Copenhagen and co-authors (LexGLUE); Stanford CRFM (HELM scenario)",
   "url": "https://github.com/coastalcph/lex-glue"
  },
  "released": "2021-10",
  "repo_url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/lex_glue_scenario.py",
  "saturation": {
   "as_of": "2022-05",
   "note": "Encoder arithmetic micro-F1 79.8 (Legal-BERT) is the published suite mean, not a HELM generation number. UNFAIR-ToS micro-F1 is already in the mid-90s for encoders. Zero-shot GPT-3.5 was 47.6% micro-F1 in Chalkidis 2023. No current HELM classic cell was read.\n",
   "status": "unknown",
   "top_score": 79.8
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM lex_glue_scenario.py (seven subsets, instructions, classification_macro_f1)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/lex_glue_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "classic_run_specs.py (lex_glue:subset=..., generation adapter, max tokens/shots)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/run_specs/classic_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "schema_classic.yaml (main_metric classification_macro_f1)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/static/schema_classic.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face LexGLUE card (CC-BY-4.0; split tables; coastalcph/lex_glue)",
    "url": "https://huggingface.co/datasets/lex_glue/raw/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face API (id coastalcph/lex_glue; created 2022-03-02)",
    "url": "https://huggingface.co/api/datasets/lex_glue"
   },
   {
    "accessed": "2026-09-08",
    "title": "coastalcph/lex-glue README (task table, encoder leaderboard, Legal-BERT 79.8/72.0)",
    "url": "https://raw.githubusercontent.com/coastalcph/lex-glue/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "LexGLUE paper HTML (Table 1 sizes; micro/macro F1 protocol)",
    "url": "https://ar5iv.labs.arxiv.org/html/2110.00976"
   },
   {
    "accessed": "2026-09-08",
    "title": "ACL 2022 anthology page (May 2022, Dublin)",
    "url": "https://aclanthology.org/2022.acl-long.297/"
   },
   {
    "accessed": "2026-09-08",
    "title": "arXiv abs (v1 2021-10-03; v4 2022-11-08)",
    "url": "https://arxiv.org/abs/2110.00976"
   },
   {
    "accessed": "2026-09-08",
    "title": "Chalkidis 2023 ChatGPT LexGLUE audit (47.6% average micro-F1)",
    "url": "https://arxiv.org/abs/2304.12202"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM README (maintenance mode 2026-06-01)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM Apache License 2.0",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/LICENSE"
   }
  ],
  "status": "active",
  "subcategory": "seven-task English legal NLU suite (ECtHR A/B, SCOTUS, EUR-LEX, LEDGAR, UNFAIR-ToS, CaseHOLD)",
  "summary": "English legal NLU suite of seven public datasets; HELM scores each subset as generation with classification_macro_f1 on test.",
  "tags": [
   "legal",
   "nlu",
   "classification",
   "helm",
   "glue-style"
  ],
  "task_format": "HELM run spec lex_glue:subset=<ecthr_a|ecthr_b|scotus|eurlex|ledgar|unfair_tos|case_hold> or subset=all. Scenario loads Hugging Face config lex_glue. Generation adapter, input noun Passage, output noun Answer. MLTC tasks use comma separated labels; CaseHOLD is QA with numbered endings.\n"
 }
}