MMLU: Professional Accounting

MMLU subject subset: Case-style problems in financial accounting, auditing, cost accounting, tax and business law.

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryknowledge
Subcategoryother
Page statusactive
Metricaccuracy
Directionhigher_is_better
Unit%
Dataset size282
Dataset licenceMIT
PublisherUC Berkeley (original); Center for AI Safety (current host)

What it measures

Case-style problems in financial accounting, auditing, cost accounting, tax and business law, mixing numeric calculation with rule application. Questions are four-option multiple-choice, drawn from the MMLU test set's "other" subcategory within the benchmark's "other" top-level group, and are graded on the single correct labelled option.

Task format

Four-option multiple-choice questions, graded on the single correct labelled option; commonly evaluated 5-shot, consistent with the rest of MMLU.

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
Claude Opus 4Anthropic72.52026-04
Claude Opus 4.6Anthropic72.52026-04
GPT-4.1OpenAI71.82026-04
Gemini 2.5 ProGoogle DeepMind71.22026-04
GPT-4oOpenAI70.52026-04
GPT-4o (2024-05-13)OpenAI70.52026-04
GPT-4o (2024-08-06)OpenAI70.52026-04
GPT-4o (2024-11-20)OpenAI70.52026-04
GPT-4o miniOpenAI70.52026-04
Claude Sonnet 4Anthropic69.82026-04
Claude Sonnet 4.5Anthropic69.82026-04
Claude Sonnet 4.5 (latest)Anthropic69.82026-04
DeepSeek R1DeepSeek68.22026-04
DeepSeek R1 0528DeepSeek68.22026-04
DeepSeek R1 0528 NVFP4 v2NVIDIA68.22026-04
DeepSeek R1 0528 Qwen3 8BDeepSeek68.22026-04
DeepSeek R1 Distill Llama 70BDeepSeek68.22026-04
DeepSeek R1 Distill Llama 8BDeepSeek68.22026-04
DeepSeek R1 Distill Qwen 1.5BDeepSeek68.22026-04
DeepSeek R1 Distill Qwen 14BDeepSeek68.22026-04
DeepSeek R1 Distill Qwen 32BDeepSeek68.22026-04
DeepSeek R1 Distill Qwen 7BDeepSeek68.22026-04
DeepSeek ReasonerDeepSeek68.22026-04
Qwen 3 235B InstructCerebras67.52026-04
Qwen3 235B-A22BAlibaba / Qwen Team67.52026-04
Mixtral 8x22B Instruct v0.1Mistral AI66.32026-04
Mistral Large (latest)Mistral AI65.82026-04
Mistral Large 2.1Mistral AI65.82026-04
Mistral Large 3Mistral AI65.82026-04
Gemma 4 31BGoogle DeepMind65.22026-04
gemma 4 31B itGoogle DeepMind65.22026-04
gemma 4 31B it GGUFUnsloth65.22026-04
Gemma 4 31B IT NVFP4NVIDIA65.22026-04
Nous Hermes 2 Yi 34BNous Research64.92024-07
Gemma 4 26BGoogle DeepMind64.52026-04
Yi 1.5 34B 32K01.AI64.22024-07
Llama 3.3 70B Instruct NVFP4NVIDIA63.82026-04
Llama-3.3-70B-InstructMeta63.82026-04
Meta Llama 3 70BMeta63.82024-07
Meta Llama 3 70B InstructMeta63.82026-04
Meta Llama 3 70B InstructNous Research63.82026-04
Yi 1.5 34B Chat01.AI63.12026-04
Yi 1.5 34B Chat 16K01.AI63.12024-07
Llama 3.1 70BMeta62.52026-04
Llama 3.1 70B InstructMeta62.52026-04
Yi 1.5 34B01.AI62.12026-04
Yi 34B Chat01.AI61.72024-07
Yi 34B 200K01.AI61.32024-07
phi 4Microsoft61.22026-04
Phi 4 mini instructMicrosoft61.22026-04
Phi 4 multimodal instructMicrosoft61.22026-04
Phi 3 mini 128K instructMicrosoft58.52024-07
Phi 3 mini 4K instructMicrosoft58.22024-07
Yi 1.5 9B Chat01.AI57.82024-07
Yi 1.5 9B Chat 16K01.AI57.82024-07
Yi 1.5 9B01.AI56.72024-07
Yi 1.5 9B 32K01.AI56.72024-07
Yi 9B01.AI56.42024-07
Mixtral 8x7B Instruct v0.1Mistral AI55.32026-04
Yi 1.5 6B01.AI54.32024-07
Meta Llama 3 8B InstructMeta52.82024-07
Meta Llama 3 8B InstructNous Research52.82024-07
Nous Hermes 2 Mixtral 8x7B DPONous Research52.82024-07
Hermes 2 Theta Llama 3 8BNous Research52.12024-07
Yi 1.5 6B Chat01.AI52.12024-07
Mixtral 8x7B v0.1Mistral AI51.82026-04
Nous Hermes 2 SOLAR 10.7BNous Research51.82024-07
Mistral 7B v0.3Mistral AI51.12024-07
mistral 7B v0.3 bnb 4bitUnsloth51.12024-07
Hermes 2 Pro Llama 3 8BNous Research50.72024-07
gemma 7B itGoogle DeepMind49.32024-07
Meta Llama 3 8BMeta49.32024-07
Meta Llama 3 8BNous Research49.32024-07
Yi 6B01.AI47.92024-07
Yi 6B Chat01.AI47.92024-07
Mistral 7B Instruct v0.2Mistral AI45.02024-07
phi 2Microsoft44.02024-07
falcon 40BTII43.62024-07
Qwen2 1.5B InstructAlibaba / Qwen Team43.62024-07
Qwen2 0.5B InstructAlibaba / Qwen Team37.22024-07
deepseek llm 7B baseDeepSeek36.52024-07
deepseek llm 7B chatDeepSeek36.52024-07
deepseek coder 6.7B baseDeepSeek34.82024-07
gemma 2BGoogle DeepMind34.02024-07
chatglm2 6BZhipu AI33.32024-07
deepseek coder 6.7B instructDeepSeek31.92024-07
gemma 2B itGoogle DeepMind30.52024-07
deepseek coder 1.3B baseDeepSeek24.52024-07
deepseek coder 1.3B instructDeepSeek24.52024-07
OLMo 1B hfAllen AI22.02024-07

Data

This page as JSON · Edit on GitHub