MMLU: Jurisprudence

Accuracy on MMLU's jurisprudence questions, one of 57 subject tests of academic and professional knowledge.

Also known as: jurisprudence

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryknowledge
SubcategoryHumanities
Page statusactive
Metricaccuracy
Directionhigher_is_better
Unit%
Dataset size108
Dataset licenceMIT
PublisherUC Berkeley

What it measures

Legal theory and the philosophy of law: schools of jurisprudence, sources of law, and the relationship between law and morality, rather than the law of any one jurisdiction. Framed as four-option multiple-choice questions and scored zero-shot or few-shot by exact match against the labelled option, as one of the 57 subject subsets that make up the MMLU benchmark.

Task format

Four-option multiple-choice question answering (A-D), one correct answer, zero-shot or few-shot.

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
Nous Hermes 2 Yi 34BNous Research89.82024-07
Yi 1.5 34B 32K01.AI88.92024-07
Yi 34B 200K01.AI88.92024-07
Yi 34B Chat01.AI88.92024-07
Meta Llama 3 70BMeta88.02024-07
Meta Llama 3 70B InstructMeta88.02026-04
Meta Llama 3 70B InstructNous Research88.02026-04
Mixtral 8x22B Instruct v0.1Mistral AI86.12026-04
Yi 1.5 34B01.AI85.22026-04
Mixtral 8x7B Instruct v0.1Mistral AI84.32026-04
Nous Hermes 2 Mixtral 8x7B DPONous Research84.32024-07
Mixtral 8x7B v0.1Mistral AI83.32026-04
Yi 1.5 34B Chat01.AI83.32026-04
Yi 1.5 34B Chat 16K01.AI83.32024-07
Yi 1.5 9B 32K01.AI80.62024-07
Claude Opus 4Anthropic80.52026-04
Claude Opus 4.6Anthropic80.52026-04
GPT-4.1OpenAI79.82026-04
Phi 3 mini 4K instructMicrosoft79.62024-07
Yi 6B01.AI79.62024-07
Yi 6B Chat01.AI79.62024-07
Yi 9B01.AI79.62024-07
Gemini 2.5 ProGoogle DeepMind79.22026-04
Nous Hermes 2 SOLAR 10.7BNous Research78.72024-07
GPT-4oOpenAI78.52026-04
GPT-4o (2024-05-13)OpenAI78.52026-04
GPT-4o (2024-08-06)OpenAI78.52026-04
GPT-4o (2024-11-20)OpenAI78.52026-04
GPT-4o miniOpenAI78.52026-04
Claude Sonnet 4Anthropic77.82026-04
Claude Sonnet 4.5Anthropic77.82026-04
Claude Sonnet 4.5 (latest)Anthropic77.82026-04
gemma 7B itGoogle DeepMind77.82024-07
Meta Llama 3 8B InstructMeta77.82024-07
Meta Llama 3 8B InstructNous Research77.82024-07
Phi 3 mini 128K instructMicrosoft77.82024-07
Yi 1.5 6B01.AI77.82024-07
Yi 1.5 6B Chat01.AI77.82024-07
Yi 1.5 9B Chat 16K01.AI77.82024-07
Hermes 2 Theta Llama 3 8BNous Research75.92024-07
Meta Llama 3 8BMeta75.02024-07
Meta Llama 3 8BNous Research75.02024-07
Yi 1.5 9B Chat01.AI75.02024-07
DeepSeek R1DeepSeek74.52026-04
DeepSeek R1 0528DeepSeek74.52026-04
DeepSeek R1 0528 NVFP4 v2NVIDIA74.52026-04
DeepSeek R1 0528 Qwen3 8BDeepSeek74.52026-04
DeepSeek R1 Distill Llama 70BDeepSeek74.52026-04
DeepSeek R1 Distill Llama 8BDeepSeek74.52026-04
DeepSeek R1 Distill Qwen 1.5BDeepSeek74.52026-04
DeepSeek R1 Distill Qwen 14BDeepSeek74.52026-04
DeepSeek R1 Distill Qwen 32BDeepSeek74.52026-04
DeepSeek R1 Distill Qwen 7BDeepSeek74.52026-04
DeepSeek ReasonerDeepSeek74.52026-04
Mistral 7B Instruct v0.2Mistral AI74.12024-07
Mistral 7B v0.3Mistral AI74.12024-07
mistral 7B v0.3 bnb 4bitUnsloth74.12024-07
Yi 1.5 9B01.AI74.12024-07
Qwen 3 235B InstructCerebras73.82026-04
Qwen3 235B-A22BAlibaba / Qwen Team73.82026-04
Hermes 2 Pro Llama 3 8BNous Research73.12024-07
phi 2Microsoft73.12024-07
Gemma 4 31BGoogle DeepMind71.52026-04
gemma 4 31B itGoogle DeepMind71.52026-04
gemma 4 31B it GGUFUnsloth71.52026-04
Gemma 4 31B IT NVFP4NVIDIA71.52026-04
Mistral Large (latest)Mistral AI71.22026-04
Mistral Large 2.1Mistral AI71.22026-04
Mistral Large 3Mistral AI71.22026-04
Gemma 4 26BGoogle DeepMind70.22026-04
Llama 3.3 70B Instruct NVFP4NVIDIA69.52026-04
Llama-3.3-70B-InstructMeta69.52026-04
falcon 40BTII69.42024-07
Qwen2 1.5B InstructAlibaba / Qwen Team69.42024-07
Llama 3.1 70BMeta68.82026-04
Llama 3.1 70B InstructMeta68.82026-04
phi 4Microsoft66.82026-04
Phi 4 mini instructMicrosoft66.82026-04
Phi 4 multimodal instructMicrosoft66.82026-04
deepseek llm 7B baseDeepSeek63.02024-07
deepseek llm 7B chatDeepSeek63.02024-07
Qwen2 0.5B InstructAlibaba / Qwen Team59.32024-07
chatglm2 6BZhipu AI58.32024-07
gemma 2B itGoogle DeepMind46.32024-07
gemma 2BGoogle DeepMind41.72024-07
deepseek coder 6.7B instructDeepSeek37.02024-07
deepseek coder 6.7B baseDeepSeek34.32024-07
deepseek coder 1.3B baseDeepSeek25.02024-07
deepseek coder 1.3B instructDeepSeek25.02024-07
OLMo 1B hfAllen AI24.12024-07

Data

This page as JSON · Edit on GitHub