LEXTREME

Multilingual legal suite of 11 datasets / 18 HELM tasks; paper aggregate 61.3 for XLM-R large, later GitHub table higher for legal-adapted encoders.

Also known as: Lextreme, Multilingual Legal Benchmark for Natural Language Understanding

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	domain
Subcategory	multilingual legal NLU (11 datasets, 18 HELM tasks, 24 EU languages in the paper)
Page status	active
Metric	classification_macro_f1 (HELM schema); paper uses dataset and language aggregate scores
Direction	higher_is_better
Unit	%
Dataset licence	CC-BY-4.0 on the Hugging Face card; arXiv HTML for 2301.13126v3 is CC BY-NC-SA 4.0
Publisher	University of Bern / Bern University of Applied Sciences / Stanford (Niklaus) and co-authors; Stanford CRFM (HELM scenario)

What it measures

LEXTREME scores legal NLU across languages rather than English-only [lex_glue](lex_glue.md). HELM implements 18 configs: Brazilian judgment and unanimity, German argument mining, three Greek Legal Code levels, Swiss judgment prediction, two online ToS tasks, COVID-19 emergency events, three MultiEURLEX EuroVoc levels, and five NER sets (Greek, Romanian, Brazilian, MAPA coarse and fine). Inputs are court text, legislation, or ToS sentences. HELM is generation over those labels, not the paper's encoder fine-tunes.

Task format

HELM run spec lextreme:subset=<config> or subset=all. Dataset joelito/lextreme. Generation adapter, input noun Passage, output noun Answer. NER outputs a quoted token-label sequence. MultiEURLEX input is a dict of languages; HELM picks one language at random per example.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub