MMLU: Professional Law

MMLU subject subset: Bar-exam-style fact patterns testing US law, and by far the largest of MMLU's 57 subjects.

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryknowledge
Subcategorylaw
Page statusactive
Metricaccuracy
Directionhigher_is_better
Unit%
Dataset size1534
Dataset licenceMIT
PublisherUC Berkeley (original); Center for AI Safety (current host)

What it measures

Bar-exam-style fact patterns testing US law across contracts, torts, criminal law, property and evidence, each resolved to a single correct rule application. Questions are four-option multiple-choice, drawn from the MMLU test set's "law" subcategory within the benchmark's "humanities" top-level group, and are graded on the single correct labelled option.

Task format

Four-option multiple-choice questions, graded on the single correct labelled option; commonly evaluated 5-shot, consistent with the rest of MMLU.

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
Claude Opus 4Anthropic78.22026-04
Claude Opus 4.6Anthropic78.22026-04
GPT-4.1OpenAI77.52026-04
Gemini 2.5 ProGoogle DeepMind77.22026-04
GPT-4oOpenAI76.82026-04
GPT-4o (2024-05-13)OpenAI76.82026-04
GPT-4o (2024-08-06)OpenAI76.82026-04
GPT-4o (2024-11-20)OpenAI76.82026-04
GPT-4o miniOpenAI76.82026-04
Claude Sonnet 4Anthropic75.52026-04
Claude Sonnet 4.5Anthropic75.52026-04
Claude Sonnet 4.5 (latest)Anthropic75.52026-04
DeepSeek R1DeepSeek72.52026-04
DeepSeek R1 0528DeepSeek72.52026-04
DeepSeek R1 0528 NVFP4 v2NVIDIA72.52026-04
DeepSeek R1 0528 Qwen3 8BDeepSeek72.52026-04
DeepSeek R1 Distill Llama 70BDeepSeek72.52026-04
DeepSeek R1 Distill Llama 8BDeepSeek72.52026-04
DeepSeek R1 Distill Qwen 1.5BDeepSeek72.52026-04
DeepSeek R1 Distill Qwen 14BDeepSeek72.52026-04
DeepSeek R1 Distill Qwen 32BDeepSeek72.52026-04
DeepSeek R1 Distill Qwen 7BDeepSeek72.52026-04
DeepSeek ReasonerDeepSeek72.52026-04
Qwen 3 235B InstructCerebras71.82026-04
Qwen3 235B-A22BAlibaba / Qwen Team71.82026-04
Mistral Large (latest)Mistral AI70.52026-04
Mistral Large 2.1Mistral AI70.52026-04
Mistral Large 3Mistral AI70.52026-04
Gemma 4 31BGoogle DeepMind70.22026-04
gemma 4 31B itGoogle DeepMind70.22026-04
gemma 4 31B it GGUFUnsloth70.22026-04
Gemma 4 31B IT NVFP4NVIDIA70.22026-04
Gemma 4 26BGoogle DeepMind69.22026-04
Llama 3.3 70B Instruct NVFP4NVIDIA68.22026-04
Llama-3.3-70B-InstructMeta68.22026-04
Llama 3.1 70BMeta67.52026-04
Llama 3.1 70B InstructMeta67.52026-04
phi 4Microsoft65.52026-04
Phi 4 mini instructMicrosoft65.52026-04
Phi 4 multimodal instructMicrosoft65.52026-04
Meta Llama 3 70BMeta64.12024-07
Meta Llama 3 70B InstructMeta64.12026-04
Meta Llama 3 70B InstructNous Research64.12026-04
Nous Hermes 2 Yi 34BNous Research61.72024-07
Mixtral 8x22B Instruct v0.1Mistral AI60.32026-04
Yi 34B 200K01.AI59.12024-07
Yi 1.5 34B 32K01.AI58.52024-07
Yi 1.5 34B01.AI58.02026-04
Yi 1.5 34B Chat01.AI57.02026-04
Yi 1.5 34B Chat 16K01.AI57.02024-07
Nous Hermes 2 Mixtral 8x7B DPONous Research55.52024-07
Yi 34B Chat01.AI55.02024-07
Mixtral 8x7B Instruct v0.1Mistral AI54.42026-04
Mixtral 8x7B v0.1Mistral AI53.22026-04
Phi 3 mini 4K instructMicrosoft51.32024-07
Yi 1.5 9B 32K01.AI50.72024-07
Yi 1.5 9B01.AI50.32024-07
Nous Hermes 2 SOLAR 10.7BNous Research50.12024-07
Yi 1.5 9B Chat 16K01.AI50.12024-07
Phi 3 mini 128K instructMicrosoft49.72024-07
Yi 6B01.AI49.52024-07
Yi 6B Chat01.AI49.52024-07
Yi 9B01.AI49.52024-07
Hermes 2 Theta Llama 3 8BNous Research48.52024-07
gemma 7B itGoogle DeepMind48.12024-07
Yi 1.5 9B Chat01.AI48.02024-07
Meta Llama 3 8B InstructMeta47.82024-07
Meta Llama 3 8B InstructNous Research47.82024-07
Yi 1.5 6B Chat01.AI47.32024-07
Yi 1.5 6B01.AI47.12024-07
Hermes 2 Pro Llama 3 8BNous Research46.82024-07
Meta Llama 3 8BMeta46.72024-07
Meta Llama 3 8BNous Research46.72024-07
Mistral 7B v0.3Mistral AI46.22024-07
mistral 7B v0.3 bnb 4bitUnsloth46.22024-07
phi 2Microsoft43.72024-07
Mistral 7B Instruct v0.2Mistral AI43.52024-07
falcon 40BTII43.22024-07
Qwen2 1.5B InstructAlibaba / Qwen Team42.82024-07
deepseek llm 7B baseDeepSeek40.72024-07
deepseek llm 7B chatDeepSeek40.72024-07
Qwen2 0.5B InstructAlibaba / Qwen Team36.22024-07
chatglm2 6BZhipu AI35.62024-07
gemma 2BGoogle DeepMind34.72024-07
gemma 2B itGoogle DeepMind31.92024-07
deepseek coder 1.3B baseDeepSeek29.02024-07
deepseek coder 1.3B instructDeepSeek29.02024-07
deepseek coder 6.7B baseDeepSeek28.82024-07
deepseek coder 6.7B instructDeepSeek28.42024-07
OLMo 1B hfAllen AI24.12024-07

Data

This page as JSON · Edit on GitHub