GSM8K

8.5K grade-school math word problems scored by exact match on the final numeric answer, testing multi-step arithmetic reasoning.

Also known as: Grade School Math 8K

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categorymath
Subcategorygrade-school arithmetic word problems
Page statusactive
Metricaccuracy (exact-match final answer)
Directionhigher_is_better
Unit%
Dataset size8792
Dataset licenceMIT
PublisherOpenAI

What it measures

GSM8K tests whether a model can solve short, English-language math word problems that take two to eight linked arithmetic steps to reach an answer, such as working out a total after several purchases and discounts. OpenAI wrote the problems so that a bright middle-school student could solve every one of them by hand, deliberately excluding algebra, calculus or other advanced mathematics. The skill under test is multi-step arithmetic planning and consistent calculation rather than mathematical sophistication.

Task format

Free-response word problem in English; the model produces a reasoning chain ending in one final numeric answer, extracted after a `####` marker and graded by exact match.

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
Kimi K2 0711Moonshot AI97.32026-04
Kimi K2 0905Moonshot AI97.32026-04
Kimi K2 ThinkingMoonshot AI97.32026-04
Kimi K2 Thinking TurboMoonshot AI97.32026-04
Kimi K2 TurboMoonshot AI97.32026-04
Kimi K2.5Moonshot AI97.32026-04
Kimi K2.5 NVFP4NVIDIA97.32026-04
o1OpenAI97.12026-04
o1-miniOpenAI97.12026-04
o1-proOpenAI97.12026-04
Llama 3.1 405BMeta96.82026-04
Llama 3.1 405B InstructMeta96.82026-04
Llama 3.1 405B Instruct FP8Meta96.82026-04
Claude Sonnet 3.5Anthropic96.42026-04
Claude Sonnet 3.5 v2Anthropic96.42026-04
Gemma 3 27BGoogle DeepMind95.92026-04
Claude Opus 3Anthropic95.02026-04
GPT-4oOpenAI95.02026-04
GPT-4o (2024-05-13)OpenAI95.02026-04
GPT-4o (2024-08-06)OpenAI95.02026-04
GPT-4o (2024-11-20)OpenAI95.02026-04
Llama 3.3 70B Instruct NVFP4NVIDIA95.02026-04
Llama-3.3-70B-InstructMeta95.02026-04
Meta Llama 3 70B InstructMeta95.02026-04
Meta Llama 3 70B InstructNous Research95.02026-04
Gemma 3 12BGoogle DeepMind94.42026-04
Qwen3 235B-A22BAlibaba / Qwen Team94.42026-04
phi 4Microsoft93.52026-04
GPT-4 TurboOpenAI93.02026-04
Mistral Large (latest)Mistral AI93.02026-04
Claude Sonnet 3Anthropic92.32026-04
GPT-4OpenAI92.02026-04
GPT-4o miniOpenAI92.02026-04
Qwen2.5 7BAlibaba / Qwen Team91.62026-04
Gemini 1.5 ProGoogle DeepMind90.82026-04
Gemini 2.5 ProGoogle DeepMind90.82026-04
DeepSeek ChatDeepSeek89.32026-04
DeepSeek ReasonerDeepSeek89.32026-04
DeepSeek V3DeepSeek89.32026-04
DeepSeek V3 0324DeepSeek89.32026-04
DeepSeek V3.1DeepSeek89.32026-04
DeepSeek V3.2DeepSeek89.32026-04
DeepSeek V3.2 ExpDeepSeek89.32026-04
Gemma 3 4BGoogle DeepMind89.22026-04
gemma 3 4B ptGoogle DeepMind89.22026-04
Claude Haiku 3Anthropic88.92026-04
Phi 4 mini instructMicrosoft88.62026-04
Mixtral 8x22BMistral AI88.02026-04
Mixtral 8x22B Instruct v0.1Mistral AI88.02026-04
Gemini 1.5 FlashGoogle DeepMind86.22026-04
Gemini 1.5 Flash-8BGoogle DeepMind86.22026-04
Gemini 2.0 FlashGoogle DeepMind86.22026-04
Gemini 2.5 FlashGoogle DeepMind86.22026-04
Phi 3.5 mini instructMicrosoft86.22026-04
Meta Llama 3 70BMeta85.42024-07
granite 3.3 8B instructIBM80.92026-04
GPT-3.5-turboOpenAI80.02026-04
Llama 3.2 3BMeta77.72026-04
Llama 3.2 3B InstructMeta77.72026-04
Phi 3 mini 4K instructMicrosoft74.52024-07
gemma 2 27B itGoogle DeepMind74.02026-04
Yi 1.5 34B01.AI73.22026-04
Yi 1.5 9B Chat01.AI71.92024-07
Nous Hermes 2 Mixtral 8x7B DPONous Research71.62024-07
Yi 1.5 34B Chat01.AI71.62026-04
Yi 1.5 34B Chat 16K01.AI71.62024-07
Command R+Cohere70.72026-04
Hermes 2 Theta Llama 3 8BNous Research70.42024-07
Nous Hermes 2 Yi 34BNous Research70.12024-07
Phi 3 mini 128K instructMicrosoft69.52024-07
Nous Hermes 2 SOLAR 10.7BNous Research69.42024-07
Meta Llama 3 8B InstructMeta68.72024-07
Meta Llama 3 8B InstructNous Research68.72024-07
gemma 2 9BGoogle DeepMind68.62026-04
gemma 2 9B itGoogle DeepMind68.62026-04
Hermes 2 Pro Llama 3 8BNous Research67.92024-07
Yi 1.5 6B Chat01.AI67.12024-07
gemma 3 1B itGoogle DeepMind62.82026-04
gemma 3 1B ptGoogle DeepMind62.82026-04
Yi 1.5 9B01.AI62.72024-07
Mixtral 8x7B Instruct v0.1Mistral AI61.12026-04
Yi 1.5 9B Chat 16K01.AI59.72024-07
Mixtral 8x7B v0.1Mistral AI57.62026-04
Qwen2 1.5B InstructAlibaba / Qwen Team55.82024-07
phi 2Microsoft55.02024-07
Llama 2 70B hfMeta54.12024-07
gemma 7B itGoogle DeepMind52.82024-07
Yi 34B01.AI50.62024-07
Yi 1.5 6B01.AI49.82024-07
Yi 9B01.AI49.02024-07
deepseek llm 7B baseDeepSeek45.92024-07
deepseek llm 7B chatDeepSeek45.92024-07
Meta Llama 3 8BMeta45.22024-07
Meta Llama 3 8BNous Research45.22024-07
Mistral 7B Instruct v0.2Mistral AI40.02024-07
Mistral 7B v0.1Mistral AI37.12024-07
Qwen2 0.5B InstructAlibaba / Qwen Team35.72024-07
Yi 34B 200K01.AI34.92024-07
Mistral 7B v0.3Mistral AI34.52024-07
mistral 7B v0.3 bnb 4bitUnsloth34.52024-07
falcon 40B instructTII34.32024-07
Yi 34B Chat01.AI31.92024-07
Yi 6B 200K01.AI30.32024-07
deepseek coder 6.7B instructDeepSeek26.82024-07
Llama 2 70B chat hfMeta26.72024-07
Llama 2 13B hfMeta22.82024-07
Llama 2 13B hfNous Research22.82024-07
CodeLlama 34B Instruct hfMeta21.62026-04
deepseek coder 6.7B baseDeepSeek18.02024-07
gemma 2BGoogle DeepMind16.92024-07
Llama 2 13B chat hfMeta15.22024-07
Llama 2 7B hfMeta14.52024-07
Llama 2 7B hfNous Research14.52024-07
Mistral 7B Instruct v0.1Mistral AI14.32024-07
Yi 6B01.AI12.12024-07
Yi 6B Chat01.AI12.12024-07
Llama 2 7B chat hfMeta7.42024-07
Llama 2 7B chat hfNous Research7.42024-07
Nous Hermes llama 2 7BNous Research5.82024-07
CodeLlama 7B hfMeta5.52024-07
CodeLlama 7B Instruct hfMeta5.52024-07
gemma 2B itGoogle DeepMind5.52024-07
falcon 7B instructTII4.72024-07
falcon 7BTII4.62024-07
Baichuan 7BBaichuan2.82024-07
OLMo 1B hfAllen AI1.92024-07
deepseek coder 1.3B baseDeepSeek1.12024-07
deepseek coder 1.3B instructDeepSeek1.12024-07
Yi 1.5 34B 32K01.AI0.02024-07
Yi 1.5 9B 32K01.AI0.02024-07

Data

This page as JSON · Edit on GitHub