LegalBench

A collaboratively built suite of 162 tasks, contributed by lawyers and computer scientists, testing six categories of legal reasoning in language models.

Also known as: LEGALBENCH

unassessed

What it measures

LegalBench tests whether a model can perform the kinds of reasoning lawyers actually use, rather than treating "legal reasoning" as one undifferentiated skill. Its 162 tasks are organised into six categories drawn from how the legal profession itself frames reasoning: issue-spotting (does a fact pattern raise a given legal question), rule-recall (state or identify the applicable rule), rule-application (explain how a rule applies to facts, with reasoning graded for correctness and analysis), rule-conclusion (state the resulting legal outcome), interpretation (parse a contract, privacy policy or statute), and rhetorical-understanding (reason about legal argument and judicial writing). Inputs are short clauses, fact patterns, questions or case excerpts; most tasks are single-turn, English-only, and text-only.

Task format

Varies by task: multiple-choice (35 tasks), open-ended generation (7), binary classification (112), and multi-class or multi-label classification (8). Few-shot prompting is standard, using 0 to 8 in-context demonstrations drawn from each task's own small training split.

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub