MMLU: Elementary Mathematics

MMLU subject subset: Grade-school arithmetic and pre-algebra, including order-of-operations and word problems.

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryknowledge
Subcategorymath
Page statusactive
Metricaccuracy
Directionhigher_is_better
Unit%
Dataset size378
Dataset licenceMIT
PublisherUC Berkeley (original); Center for AI Safety (current host)

What it measures

Grade-school arithmetic and pre-algebra, including order-of-operations and word problems. Questions are four-option multiple-choice, drawn from the MMLU test set's "math" subcategory within the benchmark's "STEM" top-level group, and are graded on the single correct labelled option.

Task format

Four-option multiple-choice questions, graded on the single correct labelled option; commonly evaluated 5-shot, consistent with the rest of MMLU.

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
Yi 1.5 34B Chat01.AI75.42026-04
Yi 1.5 34B Chat 16K01.AI75.42024-07
Yi 1.5 34B 32K01.AI74.32024-07
Yi 1.5 34B01.AI72.02026-04
Meta Llama 3 70BMeta70.92024-07
Meta Llama 3 70B InstructMeta70.92026-04
Meta Llama 3 70B InstructNous Research70.92026-04
Nous Hermes 2 Yi 34BNous Research69.02024-07
Yi 34B 200K01.AI64.62024-07
Yi 1.5 9B Chat 16K01.AI64.32024-07
Yi 1.5 9B Chat01.AI64.02024-07
Yi 34B Chat01.AI63.52024-07
Yi 1.5 9B01.AI63.22024-07
Mixtral 8x22B Instruct v0.1Mistral AI62.42026-04
Yi 1.5 9B 32K01.AI60.32024-07
Yi 9B01.AI60.12024-07
Yi 1.5 6B Chat01.AI53.42024-07
Phi 3 mini 4K instructMicrosoft52.62024-07
Yi 1.5 6B01.AI51.92024-07
gemma 7B itGoogle DeepMind50.32024-07
Nous Hermes 2 Mixtral 8x7B DPONous Research50.32024-07
Nous Hermes 2 SOLAR 10.7BNous Research48.92024-07
Phi 3 mini 128K instructMicrosoft48.92024-07
Mixtral 8x7B v0.1Mistral AI48.12026-04
Mixtral 8x7B Instruct v0.1Mistral AI47.92026-04
phi 2Microsoft46.02024-07
Hermes 2 Theta Llama 3 8BNous Research45.22024-07
Yi 6B01.AI45.02024-07
Yi 6B Chat01.AI45.02024-07
Meta Llama 3 8B InstructMeta44.42024-07
Meta Llama 3 8B InstructNous Research44.42024-07
Meta Llama 3 8BMeta43.42024-07
Meta Llama 3 8BNous Research43.42024-07
Hermes 2 Pro Llama 3 8BNous Research42.92024-07
Qwen2 1.5B InstructAlibaba / Qwen Team42.32024-07
Mistral 7B v0.3Mistral AI40.22024-07
mistral 7B v0.3 bnb 4bitUnsloth40.22024-07
Mistral 7B Instruct v0.2Mistral AI36.82024-07
Qwen2 0.5B InstructAlibaba / Qwen Team35.22024-07
deepseek coder 6.7B instructDeepSeek34.12024-07
falcon 40BTII33.92024-07
chatglm2 6BZhipu AI31.72024-07
deepseek coder 6.7B baseDeepSeek31.22024-07
deepseek llm 7B baseDeepSeek31.22024-07
deepseek llm 7B chatDeepSeek31.22024-07
gemma 2BGoogle DeepMind26.22024-07
deepseek coder 1.3B baseDeepSeek25.92024-07
deepseek coder 1.3B instructDeepSeek25.92024-07
gemma 2B itGoogle DeepMind24.92024-07
OLMo 1B hfAllen AI24.62024-07

Data

This page as JSON · Edit on GitHub