{
 "body": "\n## What it measures\n\nLEXTREME tests legal classification and NER in many European languages.\nA model may see a Brazilian appeal, German Urteilsstil sentences, Greek\nlegislation, Swiss Federal Supreme Court facts, a consumer ToS clause, a\nCOVID-era statute sentence, an EU act, or a NER sentence from Greek,\nRomanian, Brazilian, or EUR-Lex text. Outputs are topic or judgment labels,\nunfair-clause tags, or per-token entity tags. The paper\u2019s point is that\nEnglish-only legal suites such as [lex_glue](lex_glue.md) miss multilingual\nlegal systems. HELM does not implement the Ukrainian court-decision configs\nthat the 2026 Hub card added.\n\n## How it is scored\n\nNiklaus et al. fine-tune multilingual encoders and report a dataset\naggregate and a language aggregate, each a nested harmonic mean. In the\npaper, XLM-R large scores 61.3 on both. The project README later prints\ndifferent encoder aggregates (for example XLM-R-large 66.1 dataset / 53.7\nlanguage, Legal-XLM-LF-base 66.9 dataset). HELM classic instead headlines\n`classification_macro_f1` on generation runs `lextreme:subset=...`. Shot\nlimits are 1\u20135 and max tokens range from 5 to several hundred for NER.\nMultiEURLEX examples keep one randomly chosen language. A HELM number is\nnot the paper aggregate.\n\n## Dataset and licence\n\nHugging Face `joelito/lextreme` (API id `joelniklaus/lextreme`) is the\nbuilder HELM loads. The card licence is CC-BY-4.0. The arXiv v3 HTML\nlicence line is CC BY-NC-SA 4.0. Source datasets keep their own terms; the\npaper excluded sets without a redistributable licence. Per-config sizes\nare in the GitHub README (for example Swiss judgment test 17,357,\nMultiEURLEX test 115,000 documents). Do not sum those tests into one n.\nLabels are public.\n\n## Who publishes it\n\nJoel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias St\u00fcrmer,\nand Ilias Chalkidis released the suite on arXiv on 30 January 2023 and at\nEMNLP 2023 Findings. Code and encoder tables are in JoelNiklaus/LEXTREME.\nStanford CRFM added the HELM classic scenario. HELM entered maintenance\nmode on 2026-06-01.\n\n## Lineage\n\nPredecessor: [lex_glue](lex_glue.md) as the English legal multi-task\ntemplate. Several LEXTREME sources overlap that world (ToS unfairness,\nEUR-Lex labels) but the languages and configs differ. No successor id is\nin this repository. [legalbench](legalbench.md) is English LLM reasoning,\nnot this encoder suite.\n\n## Saturation and contamination\n\nEven the best published encoder aggregates sit in the 60s, so the suite is\nstill open on that protocol. Generative HELM saturation was not read.\nSource corpora are old and public, so contamination risk is high.\n\n## How to run it\n\nPaper reproduction: `python main.py` in JoelNiklaus/LEXTREME (encoder\nfine-tunes, optional W&B). HELM: `lextreme:subset=swiss_judgment_prediction`\nor another key from `TASK_CODE_MAPPING`. NER formatting uses a quoted\n`\"token\" \"token\"` delimiter. MultiEURLEX scores will move if the language\nsampler seed changes. Do not drop a Ukrainian config into HELM without new\ncode.\n\n## Reading the numbers\n\nA strong dataset aggregate means the model is decent across these legal\ntasks after you fold languages inside each dataset. A strong language\naggregate means it does not only win on Portuguese or English. A HELM\ngeneration F1 on one subset is neither of those headlines. The suite does\nnot score contract drafting, retrieval, or US bar exams. Prefer the paper\naggregates for encoder claims and named HELM subsets for generative claims,\nand say which of the two XLM-R-large tables you used.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "lextreme",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Lextreme",
   "Multilingual Legal Benchmark for Natural Language Understanding"
  ],
  "category": "domain",
  "contamination": {
   "note": "Every source dataset was already public. The Hub builder joelniklaus/lextreme (card also served as joelito/lextreme) has been public since 2022-08-01, with a 2026-05-20 card update. HELM loads labels with the text.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "bg",
    "cs",
    "da",
    "de",
    "el",
    "en",
    "es",
    "et",
    "fi",
    "fr",
    "ga",
    "hr",
    "hu",
    "it",
    "lt",
    "lv",
    "mt",
    "nl",
    "pl",
    "pt",
    "ro",
    "sk",
    "sl",
    "sv",
    "uk"
   ],
   "license": "CC-BY-4.0 on the Hugging Face card; arXiv HTML for 2301.13126v3 is CC BY-NC-SA 4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "Do not add the 18 test splits: MultiEURLEX level 1/2/3 share the same documents (train 817,239 / val 112,500 / test 115,000 in the project README). Other test n from that README include SJP 17,357, GLC 9,516, GAM 3,078, MAPA 10,590, BCD-J 405. The 2023 paper selects 11 datasets and 24 languages. The Hugging Face card (lastModified 2026-05-20) now lists a 12th dataset, Ukrainian court decisions, and language uk; HELM's TASK_CODE_MAPPING still has the original 18 configs.\n",
   "splits": "per config train/validation/test as published or 80/10/10 when the source had none; HELM scores test",
   "url": "https://huggingface.co/datasets/joelito/lextreme"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-053 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-053"
  },
  "harness": {
   "bigbench": "",
   "helm": "lextreme:subset=<task>",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Group lextreme. Dataset joelito/lextreme. 18 HELM configs; Ukrainian court decisions are on the Hub card but not in HELM's mapping."
  },
  "id": "lextreme",
  "last_updated": "2026-05",
  "leaderboard_url": "https://github.com/JoelNiklaus/LEXTREME",
  "lineage": {
   "family": "",
   "predecessor": "lex_glue",
   "successors": [],
   "variants": []
  },
  "measures": "LEXTREME scores legal NLU across languages rather than English-only [lex_glue](lex_glue.md). HELM implements 18 configs: Brazilian judgment and unanimity, German argument mining, three Greek Legal Code levels, Swiss judgment prediction, two online ToS tasks, COVID-19 emergency events, three MultiEURLEX EuroVoc levels, and five NER sets (Greek, Romanian, Brazilian, MAPA coarse and fine). Inputs are court text, legislation, or ToS sentences. HELM is generation over those labels, not the paper's encoder fine-tunes.\n",
  "metric": {
   "baseline_note": "schema_classic.yaml main_name classification_macro_f1, main_split test. The paper's headline is two harmonic-mean aggregates (dataset and language), both 61.3 for XLM-R large. The GitHub README later table lists XLM-R-large dataset aggregate 66.1 and language aggregate 53.7, with Legal-XLM-LF-base 66.9 dataset aggregate. Those encoder numbers are not HELM generation scores. NER is a different output space.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "classification_macro_f1 (HELM schema); paper uses dataset and language aggregate scores",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "LEXTREME",
  "page_kind": "family",
  "paper": {
   "arxiv": "2301.13126",
   "title": "LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain",
   "url": "https://aclanthology.org/2023.findings-emnlp.200/",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Joel Niklaus",
    "Veton Matoshi",
    "Pooja Rani",
    "Andrea Galassi",
    "Matthias St\u00fcrmer",
    "Ilias Chalkidis"
   ],
   "org": "University of Bern / Bern University of Applied Sciences / Stanford (Niklaus) and co-authors; Stanford CRFM (HELM scenario)",
   "url": "https://github.com/JoelNiklaus/LEXTREME"
  },
  "released": "2023-01",
  "repo_url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/lextreme_scenario.py",
  "saturation": {
   "as_of": "2023-01",
   "note": "Paper: XLM-R large dataset and language aggregates both 61.3. GitHub README later reports higher dataset aggregates (Legal-XLM-LF-base 66.9) and a 53.7 language aggregate for XLM-R-large. Either way the suite is far from 100. ChatGPT is described as struggling. No HELM generation top score was read.\n",
   "status": "open",
   "top_score": 61.3
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM lextreme_scenario.py (18 configs, instructions, NER formatting, MultiEURLEX language sample)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/lextreme_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "classic_run_specs.py (lextreme:subset=..., generation adapter)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/run_specs/classic_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "schema_classic.yaml (classification_macro_f1)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/static/schema_classic.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face LEXTREME card (CC-BY-4.0; 12 datasets / 21 tasks including Ukrainian)",
    "url": "https://huggingface.co/datasets/joelito/lextreme/raw/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face API (id joelniklaus/lextreme; lastModified 2026-05-20; 25 languages including uk)",
    "url": "https://huggingface.co/api/datasets/joelito/lextreme"
   },
   {
    "accessed": "2026-09-08",
    "title": "JoelNiklaus/LEXTREME README (18-task table, later encoder aggregates)",
    "url": "https://raw.githubusercontent.com/JoelNiklaus/LEXTREME/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "LEXTREME paper HTML (11 datasets, 24 languages, XLM-R large 61.3)",
    "url": "https://ar5iv.labs.arxiv.org/html/2301.13126"
   },
   {
    "accessed": "2026-09-08",
    "title": "arXiv HTML v3 (license line CC BY-NC-SA 4.0; 8 Jan 2024)",
    "url": "https://arxiv.org/html/2301.13126v3"
   },
   {
    "accessed": "2026-09-08",
    "title": "EMNLP 2023 Findings anthology page",
    "url": "https://aclanthology.org/2023.findings-emnlp.200/"
   },
   {
    "accessed": "2026-09-08",
    "title": "arXiv abs (v1 2023-01-30; v3 2024-01-08)",
    "url": "https://arxiv.org/abs/2301.13126"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM README (maintenance mode 2026-06-01)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM Apache License 2.0",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/LICENSE"
   }
  ],
  "status": "active",
  "subcategory": "multilingual legal NLU (11 datasets, 18 HELM tasks, 24 EU languages in the paper)",
  "summary": "Multilingual legal suite of 11 datasets / 18 HELM tasks; paper aggregate 61.3 for XLM-R large, later GitHub table higher for legal-adapted encoders.",
  "tags": [
   "legal",
   "multilingual",
   "nlu",
   "ner",
   "helm"
  ],
  "task_format": "HELM run spec lextreme:subset=<config> or subset=all. Dataset joelito/lextreme. Generation adapter, input noun Passage, output noun Answer. NER outputs a quoted token-label sequence. MultiEURLEX input is a dict of languages; HELM picks one language at random per example.\n"
 }
}