{
 "body": "\n## What it measures\n\nLegalBench tests whether a model can perform the specific kinds of reasoning lawyers use, rather\nthan treating \"legal reasoning\" as one skill. Its 162 tasks are organised into six categories drawn\ndirectly from how the legal profession frames reasoning, following the IRAC framework taught in\nAmerican law schools: issue-spotting (does a fact pattern raise a given legal question),\nrule-recall (state or identify the applicable rule), rule-application (explain how a rule applies to\na set of facts), rule-conclusion (state the resulting legal outcome), plus two categories outside\nIRAC \u2014 interpretation (parse a contract, privacy policy or statute) and rhetorical-understanding\n(reason about legal argument and judicial writing). Inputs are short clauses, fact patterns,\nquestions or case excerpts, drawn from areas including contracts, privacy policy, evidence and\ncorporate and tax law; the benchmark is English-only and skews toward U.S. federal law.\n\n## How it is scored\n\nMost of the 162 tasks are automatically scored: classification tasks use balanced accuracy (many\nare class-imbalanced), and multi-answer extraction tasks use F1. A handful of generation tasks use\ncustom rules, such as accepting a tax-amount prediction within 10% of the true value. The 12\nrule-application tasks are the exception: a law-trained annotator manually grades each model\nexplanation against a published answer guide, on two separate axes \u2014 whether it is factually and\nlegally correct, and whether it actually contains analysis rather than merely restating the rule\nand the predicted outcome \u2014 because the authors found models often produce conclusory explanations\nthat are technically correct but analytically empty. There is no single official \"LegalBench score\"\ncombining all 162 tasks; the original paper reports per-category averages across 20 models.\n\n## Dataset and licence\n\nThe 162 tasks are drawn from 36 distinct source corpora \u2014 some pre-existing datasets restructured\nfor few-shot prompting (e.g. CUAD, MAUD, OPP-115), some previously unpublished data hand-coded by\nlegal scholars, and some built specifically for LegalBench by its author-contributors. In total the\ndataset holds 91,750 examples (82.5 MB), averaging 563 per task with a floor of 50. Every task\ncarries a small few-shot train split (2-8 examples) rather than a conventional large training set,\nby design, since the benchmark targets in-context learning rather than fine-tuning. Licensing is set\nper task by its original source: most are CC BY 4.0, with several others under CC BY-NC 4.0,\nCC BY-SA 4.0, CC BY-NC-SA 4.0, MIT, CC BY-NC or CC BY-NC 3.0, documented task-by-task in the paper.\n\n## Who publishes it\n\nLegalBench was introduced by a 40-author collaboration led by Neel Guha, Julian Nyarko, Daniel E.\nHo and Christopher R\u00e9 (credited as equal contributors), together with Adam Chilton and 35 further\nco-authors spanning Stanford, the University of Chicago, Dartmouth, Harvard Law School, the\nUniversity of Toronto, Georgetown University Law Center and other law schools and computer science\ndepartments, plus practicing lawyers. The paper, posted to arXiv in August 2023, describes an\nAugust 2022 open call for task contributions, publicised through legal-computing mailing lists and\nconferences, with submissions vetted for legal correctness before inclusion. Neel Guha maintains the\ndataset and the GitHub repository, and the project states it intends to keep incorporating new\ntasks contributed by the legal community.\n\n## Lineage\n\nLegalBench names no single predecessor; the paper positions itself against narrower prior legal NLP\nbenchmarks such as LexGLUE and against fine-tuning-era efforts, while explicitly citing GLUE,\nBIG-bench and HELM's task organisation as inspirations for its own fine-grained, typed structure.\nThe clearest downstream relationship is with HELM: Stanford's HELM Lite folds LegalBench in as one\nof ten core scenarios, running a 5-task sample of it (abercrombie, corporate_lobbying,\nfunction_of_decision_section, international_citizenship_questions, proa) against the models on its\nleaderboard \u2014 a genuine but partial reuse, not full-suite coverage. Neither LexGLUE nor a\nLegalBench-specific successor benchmark has a page in this repository yet.\n\n## Saturation and contamination\n\nAt release, GPT-4 led every category the paper measured (59.2% on rule-recall up to 89.9% on\nrule-conclusion) but sat well short of a ceiling, especially on rule-application analysis and on the\nhardest interpretation tasks \u2014 for example, GPT-4 scored only 47.8% on the MAUD merger-agreement\nmultiple-choice questions. HELM Lite continues to score current models against its 5-task\nLegalBench sample, but this repository found no source tracking current scores across the full\n162-task suite, so a present-day saturation call is not established here. Contamination risk is\nmedium: the benchmark is fully public with public labels, several component tasks were already\npublic before LegalBench existed, and the paper itself flags that commercial models it tested may\nalready have seen some tasks during pretraining \u2014 which is why its own appendix classifies every\ntask by public-availability status, to let readers weigh this per task.\n\n## How to run it\n\nThe authors' own repository (HazyResearch/legalbench) is the reference implementation, shipping a\nbase prompt per task and, for the 12 rule-application tasks, the manual-grading answer guide used\nin the paper. HELM Lite runs a 5-task sample under the scenario name `legalbench`, with individual\ntasks addressed as `legalbench:subset=<task>` (for example `legalbench:subset=abercrombie`). No\nlm-evaluation-harness, inspect_evals, OpenCompass or BIG-bench task list was confirmed to carry\nLegalBench. Because prompt format materially changed results in the original study \u2014 Claude-1\nneeded Anthropic-specific `<example>` tag formatting to produce usable output \u2014 and because\nfew-shot demonstration counts vary by task (0 to 8), scores from different harnesses or prompting\nsetups are not guaranteed to be comparable even on the same task.\n\n## Reading the numbers\n\nA high LegalBench score on a given category says a model is good at that specific kind of legal\nreasoning \u2014 say, spotting which area of law a fact pattern implicates \u2014 not that it is a competent\nlawyer across the board; the paper's own point is that \"legal reasoning\" fractures into distinguishable\nskills with different difficulty profiles. A single combined score across the 162 tasks is\nuninformative on its own, since it blends categories as different as multiple-choice merger-agreement\ninterpretation and multi-label privacy-policy classification; look at the category breakdown, or the\nindividual task, that matches the use case in question. Treat any rule-application \"analysis\" score\nwith particular caution unless the grading method (the original manual, answer-guide-based grading,\nor some automated substitute) is stated, since that dimension does not reduce to exact-match scoring.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "legalbench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Medium (latest)",
   "model_id": "mistral/mistral-medium-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 89.4,
   "source": "lmarena.ai, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Medium 3",
   "model_id": "mistral/mistral-medium-2505",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 89.4,
   "source": "lmarena.ai, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Medium 3.1",
   "model_id": "mistral/mistral-medium-2508",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 89.4,
   "source": "lmarena.ai, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 3.1 Pro Preview",
   "model_id": "google/gemini-3-1-pro-preview",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.4,
   "source": "anthropic-system-card-mythos, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 3 Pro Preview",
   "model_id": "google/gemini-3-pro-preview",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.0,
   "source": "livecodebench, aider, swe-bench, terminal-bench, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 3 Flash Preview",
   "model_id": "google/gemini-3-flash-preview",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 86.9,
   "source": "livecodebench, aider, swe-bench, terminal-bench, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5",
   "model_id": "openai/gpt-5",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 86.0,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4",
   "model_id": "anthropic/claude-opus-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 75.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.6",
   "model_id": "anthropic/claude-opus-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 75.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, anthropic-system-card-mythos"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro",
   "model_id": "google/gemini-2-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 73.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 05-06",
   "model_id": "google/gemini-2-5-pro-preview-05-06",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 73.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 06-05",
   "model_id": "google/gemini-2-5-pro-preview-06-05",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 73.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview TTS",
   "model_id": "google/gemini-2-5-pro-preview-tts",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 73.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o",
   "model_id": "openai/gpt-4o",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 72.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-05-13)",
   "model_id": "openai/gpt-4o-2024-05-13",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 72.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-08-06)",
   "model_id": "openai/gpt-4o-2024-08-06",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 72.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-11-20)",
   "model_id": "openai/gpt-4o-2024-11-20",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 72.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o mini",
   "model_id": "openai/gpt-4o-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 72.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 70B",
   "model_id": "meta/llama-3-1-70b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 58.5,
   "source": "lmarena.ai, provider-reports, safety-evals, domain-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 70B Instruct",
   "model_id": "meta/llama-3-1-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 58.5,
   "source": "lmarena.ai, provider-reports, safety-evals, domain-evals, open-llm-leaderboard-v2"
  }
 ],
 "page": {
  "aliases": [
   "LEGALBENCH"
  ],
  "category": "domain",
  "contamination": {
   "note": "LegalBench is fully public with public labels, and the authors' own datasheet flags the risk directly: many tasks are adapted from datasets (CUAD, MAUD, OPP-115, SARA, and others) that were already public before LegalBench existed, and the paper itself notes it is \"possible that some LEGALBENCH tasks leaked into pretraining data\" for the commercial models it tested. The appendix's \"public availability status\" table sorts tasks into previously-published, original but available, and original-and-unavailable categories specifically so readers can judge this per task rather than for the suite as a whole.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Mixed, task by task: most tasks are CC BY 4.0; a smaller number are CC BY-NC 4.0 (e.g. Canada Tax Court Outcomes, Consumer Contracts QA), CC BY-SA 4.0 (Definition tasks), CC BY-NC-SA 4.0 (Learned Hands tasks), MIT (NY Judicial Ethics, Privacy Policy QA, SARA), CC BY-NC (OPP-115) or CC BY-NC 3.0 (Privacy Policy Entailment), inherited from each task's original source dataset.\n",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 162,
   "size_note": "162 tasks drawn from 36 distinct source corpora, totalling 91,750 examples (82.5 MB on Hugging Face); task size ranges from the minimum of 50 examples up to several thousand, averaging about 563. Each task ships a small train split (2-8 examples, sized for genuine few-shot prompting) and a larger evaluation split, not a conventional large train/test partition.\n",
   "splits": "per task: a small few-shot train split (2-8 examples) plus a larger evaluation split",
   "url": "https://huggingface.co/datasets/nguha/legalbench"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 1b, slice P",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "legalbench:subset=<task> (HELM Lite core scenario; HELM Lite runs a 5-task sample \u2014 abercrombie, corporate_lobbying, function_of_decision_section, international_citizenship_questions, proa \u2014 not the full 162-task suite)",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "The authors' own repository (HazyResearch/legalbench) is the reference harness: it ships a base prompt (instructions plus demonstrations) for every task, plus the answer guide used for manual grading of rule-application tasks. Anthropic-format prompts (<example> tags) were required to get usable output from Claude-1 in the original study, which is itself a reminder that prompt format is not neutral across model families on this benchmark.\n"
  },
  "id": "legalbench",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "LegalBench tests whether a model can perform the kinds of reasoning lawyers actually use, rather than treating \"legal reasoning\" as one undifferentiated skill. Its 162 tasks are organised into six categories drawn from how the legal profession itself frames reasoning: issue-spotting (does a fact pattern raise a given legal question), rule-recall (state or identify the applicable rule), rule-application (explain how a rule applies to facts, with reasoning graded for correctness and analysis), rule-conclusion (state the resulting legal outcome), interpretation (parse a contract, privacy policy or statute), and rhetorical-understanding (reason about legal argument and judicial writing). Inputs are short clauses, fact patterns, questions or case excerpts; most tasks are single-turn, English-only, and text-only.\n",
  "metric": {
   "baseline_note": "Classification tasks are scored by exact-match, using balanced accuracy because many tasks are class-imbalanced. Multi-answer extraction tasks (e.g. identifying several defendants) use F1. A handful of generation tasks use custom rules (e.g. sara_numeric counts a prediction correct if within 10% of the true tax amount). The 12 rule-application tasks are graded manually by a law-trained annotator against a published answer guide, on two separate axes: whether the explanation is factually and legally correct, and whether it actually contains analysis rather than just restating the rule and the outcome. There is no official random or human baseline; the original paper's own study evaluated 20 LLMs from 11 families and reports per-category averages, not one combined \"LegalBench score.\"\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy, balanced accuracy, F1, or human-graded correctness/analysis (varies by task)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "LegalBench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2308.11462",
   "title": "LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models",
   "url": "https://arxiv.org/abs/2308.11462",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Neel Guha",
    "Julian Nyarko",
    "Daniel E. Ho",
    "Christopher R\u00e9"
   ],
   "org": "Stanford University, with an interdisciplinary, 40-author collaboration spanning law schools, computer science departments and legal practice at institutions including the University of Chicago, Harvard Law School, Georgetown University Law Center and others",
   "url": "https://github.com/HazyResearch/legalbench/"
  },
  "released": "2023-08",
  "repo_url": "https://github.com/HazyResearch/legalbench/",
  "saturation": {
   "as_of": "",
   "note": "At the paper's August 2023 release, GPT-4 led every category (59.2-89.9% depending on category) but was far from a ceiling, especially on rule-application analysis and on harder interpretation tasks (e.g. GPT-4 scored 47.8% on the MAUD merger-agreement questions). HELM Lite continues to track a 5-task sample of LegalBench (abercrombie, corporate_lobbying, function_of_decision_section, international_citizenship_questions, proa) against current models, but this repository found no source that aggregates current scores across the full 162-task suite, so a present-day, whole-suite saturation call is not established here.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models",
    "url": "https://arxiv.org/abs/2308.11462"
   },
   {
    "accessed": "2026-09-08",
    "title": "HazyResearch/legalbench (GitHub repository, README)",
    "url": "https://github.com/HazyResearch/legalbench/"
   },
   {
    "accessed": "2026-09-08",
    "title": "nguha/legalbench dataset card (Hugging Face)",
    "url": "https://huggingface.co/datasets/nguha/legalbench"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM Lite leaderboard (Stanford CRFM) \u2014 core scenarios including LegalBench",
    "url": "https://crfm.stanford.edu/helm/lite/latest/"
   }
  ],
  "status": "active",
  "subcategory": "legal reasoning",
  "summary": "A collaboratively built suite of 162 tasks, contributed by lawyers and computer scientists, testing six categories of legal reasoning in language models.",
  "tags": [
   "legal",
   "reasoning",
   "multiple-choice",
   "classification",
   "few-shot",
   "collaborative-benchmark"
  ],
  "task_format": "Varies by task: multiple-choice (35 tasks), open-ended generation (7), binary classification (112), and multi-class or multi-label classification (8). Few-shot prompting is standard, using 0 to 8 in-context demonstrations drawn from each task's own small training split.\n"
 }
}