Legal summarization (HELM)

HELM group that scores English summaries of US bills, US civil-rights case writeups, and EU acts with ROUGE-2.

Also known as: billsum_legal_summarization, multilexsum_legal_summarization, eurlexsum_legal_summarization

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	domain
Subcategory	HELM English legal-document summarization (BillSum, MultiLexSum, EurLexSum)
Page status	unknown
Metric	rouge_2
Direction	higher_is_better
Dataset licence	Mixed: BillSum CC0-1.0; MultiLexSum ODC-By (summaries/metadata CC BY-NC); EurLexSum CC-BY-4.0; HELM code Apache-2.0
Publisher	Stanford CRFM (HELM); datasets from FiscalNote, Allen AI / Civil Rights Litigation Clearinghouse, and Heidelberg EUR-Lex-Sum

What it measures

legal_summarization is HELM's group over three published legal summarization corpora, not a new item set. The model receives a truncated source document and must write a summary. BillSum uses US bill text. MultiLexSum, in this harness, feeds the expert long summary and asks for the short summary, not the raw multi-document case file. EurLexSum uses the English EU legal act and its official summary. HELM currently loads only English EurLexSum. Generation, English text.

Task format

Prompt pattern in the scenario docstring: "Summarize the given document. Document: … Summary: …". Three @run_spec_function names share groups legal_summarization and summarization. Temperature default 0.3. Scenario class .name is "summarization", which is not a run spec.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub