MATH-500

A fixed 500-problem subset of the MATH test set used to grade free-response competition mathematics after the rest of the split was moved into training data.

Also known as: MATH500, hendrycks_math500, minerva_math500

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categorymath
Subcategorycompetition mathematics
Page statusactive
Metricaccuracy (pass@1)
Directionhigher_is_better
Unit%
Dataset size500
PublisherOpenAI

What it measures

MATH-500 gives a model 500 written competition mathematics problems spanning algebra, geometry, number theory, counting and probability, precalculus, intermediate algebra and prealgebra. The model reads a problem statement in English and must produce a worked solution ending in a final answer; there are no answer choices. It tests multi-step symbolic and numeric reasoning and correct execution of a solution plan, not speed, tool use, or any skill beyond English-language mathematics.

Task format

Free-response: read a competition math problem, output a worked solution and a final answer (typically after 'Answer:' or boxed).

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
DeepSeek R1 0528DeepSeek97.52026-04
DeepSeek R1 0528 NVFP4 v2NVIDIA97.52026-04
DeepSeek R1DeepSeek97.32026-04
DeepSeek ReasonerDeepSeek97.32026-04
o3-proOpenAI97.32026-04
GPT-5.1OpenAI96.82026-04
GPT-5.1 ChatOpenAI96.82026-04
GPT-5.1 CodexOpenAI96.82026-04
GPT-5.1 Codex MaxOpenAI96.82026-04
GPT-5.1 Codex miniOpenAI96.82026-04
o3OpenAI96.72026-04
o3-deep-researchOpenAI96.72026-04
Claude Mythos PreviewAnthropic96.42026-04
Claude Opus 4Anthropic96.42026-04
Claude Opus 4.6Anthropic96.42026-04
o4-miniOpenAI95.82026-04
o4-mini-deep-researchOpenAI95.82026-04
o1-proOpenAI95.62026-04
GPT-5OpenAI95.52026-04
GPT-5 Chat (latest)OpenAI95.52026-04
GPT-5 ProOpenAI95.52026-04
GPT-5-CodexOpenAI95.52026-04
GPT-5.2OpenAI95.52026-04
GPT-5.2 ChatOpenAI95.52026-04
GPT-5.2 CodexOpenAI95.52026-04
GPT-5.2 ProOpenAI95.52026-04
GPT-5.3 Chat (latest)OpenAI95.52026-04
GPT-5.3 CodexOpenAI95.52026-04
GPT-5.3 Codex SparkOpenAI95.52026-04
GPT-5.4OpenAI95.52026-04
GPT-5.4 miniOpenAI95.52026-04
GPT-5.4 nanoOpenAI95.52026-04
GPT-5.4 ProOpenAI95.52026-04
Claude Opus 4.5Anthropic95.22026-04
Claude Opus 4.5 (latest)Anthropic95.22026-04
Gemini 2.5 ProGoogle DeepMind95.22026-04
Gemini 2.5 Pro Preview 05-06Google DeepMind95.22026-04
Gemini 2.5 Pro Preview 06-05Google DeepMind95.22026-04
Gemini 2.5 Pro Preview TTSGoogle DeepMind95.22026-04
o3-miniOpenAI94.82026-04
Claude Opus 4.1Anthropic94.52026-04
Claude Opus 4.1 (latest)Anthropic94.52026-04
Grok 4xAI94.02026-04
Grok 4 FastxAI94.02026-04
Grok 4 Fast (Non-Reasoning)xAI94.02026-04
Grok 4.1 FastxAI94.02026-04
Grok 4.1 Fast (Non-Reasoning)xAI94.02026-04
Grok 4.20 (Non-Reasoning)xAI94.02026-04
Grok 4.20 (Reasoning)xAI94.02026-04
Grok 4.20 Multi-AgentxAI94.02026-04
Claude Sonnet 4Anthropic93.82026-04
Claude Sonnet 4.6Anthropic93.82026-04
Qwen3 235B-A22BAlibaba / Qwen Team93.52026-04
Claude Opus 4 (latest)Anthropic92.82026-04
DeepSeek V3.2DeepSeek92.52026-04
DeepSeek V3.2 ExpDeepSeek92.52026-04
o1OpenAI92.52026-04
o1-previewOpenAI92.52026-04
Claude Sonnet 4.5Anthropic92.12026-04
Claude Sonnet 4.5 (latest)Anthropic92.12026-04
Qwen3-Coder 480B-A35B InstructAlibaba / Qwen Team91.22026-04
DeepSeek V3.1DeepSeek91.02026-04
Gemini 2.5 FlashGoogle DeepMind90.52026-04
Gemini 2.5 Flash ImageGoogle DeepMind90.52026-04
Gemini 2.5 Flash Image (Preview)Google DeepMind90.52026-04
Gemini 2.5 Flash LiteGoogle DeepMind90.52026-04
Gemini 2.5 Flash Lite Preview 06-17Google DeepMind90.52026-04
Gemini 2.5 Flash Lite Preview 09-25Google DeepMind90.52026-04
Gemini 2.5 Flash Preview 04-17Google DeepMind90.52026-04
Gemini 2.5 Flash Preview 05-20Google DeepMind90.52026-04
Gemini 2.5 Flash Preview 09-25Google DeepMind90.52026-04
Gemini 2.5 Flash Preview TTSGoogle DeepMind90.52026-04
QwQ PlusAlibaba / Qwen Team90.22026-04
o1-miniOpenAI90.02026-04
GPT-4.1OpenAI89.42026-04
DeepSeek ChatDeepSeek89.22026-04
DeepSeek V2DeepSeek89.22026-04
DeepSeek V2 LiteDeepSeek89.22026-04
DeepSeek V2 Lite ChatDeepSeek89.22026-04
DeepSeek V3DeepSeek89.22026-04
DeepSeek V3 0324DeepSeek89.22026-04
GPT-5 MiniOpenAI89.12026-04
Claude Sonnet 4 (latest)Anthropic88.62026-04
DeepSeek R1 Distill Llama 70BDeepSeek88.52026-04
Qwen3 32BAlibaba / Qwen Team88.52026-04
Qwen3 32B AWQAlibaba / Qwen Team88.52026-04
Qwen3 32B NVFP4NVIDIA88.52026-04
Gemma 4 31BGoogle DeepMind87.32026-04
gemma 4 31B itGoogle DeepMind87.32026-04
gemma 4 31B it GGUFUnsloth87.32026-04
Gemma 4 31B IT NVFP4NVIDIA87.32026-04
DeepSeek R1 Distill Qwen 32BDeepSeek86.22026-04
Claude Sonnet 3.7Anthropic85.52026-04
Magistral Medium (latest)Mistral AI85.52026-04
Gemma 4 26BGoogle DeepMind85.12026-04
gemma 4 26B A4B itGoogle DeepMind85.12026-04
gemma 4 26B A4B it GGUFUnsloth85.12026-04
Claude Haiku 4.5Anthropic84.32026-04
Claude Haiku 4.5 (latest)Anthropic84.32026-04
Qwen3 30B A3B Instruct 2507Alibaba / Qwen Team84.22026-04
Qwen3 30B A3B NVFP4NVIDIA84.22026-04
Qwen3 30B-A3BAlibaba / Qwen Team84.22026-04
Qwen2.5 72B InstructAlibaba / Qwen Team83.22026-04
Grok 3xAI82.52026-04
Grok 3 FastxAI82.52026-04
Grok 3 Fast LatestxAI82.52026-04
Grok 3 LatestxAI82.52026-04
phi 4Microsoft82.52026-04
Phi 4 multimodal instructMicrosoft82.52026-04
Gemini 2.0 FlashGoogle DeepMind82.32026-04
GPT-4.1 miniOpenAI82.32026-04
Llama 4 Maverick 17B 128E InstructMeta82.22026-04
Llama-4-Maverick-17B-128E-Instruct-FP8Meta82.22026-04
DeepSeek R1 Distill Qwen 14BDeepSeek82.12026-04
Grok 3 MinixAI80.52026-04
Grok 3 Mini FastxAI80.52026-04
Grok 3 Mini Fast LatestxAI80.52026-04
Grok 3 Mini LatestxAI80.52026-04
Qwen3 14BAlibaba / Qwen Team80.52026-04
Qwen3 14B AWQAlibaba / Qwen Team80.52026-04
Qwen3 14B NVFP4NVIDIA80.52026-04
Mistral Large (latest)Mistral AI78.52026-04
Mistral Large 2.1Mistral AI78.52026-04
Mistral Large 3Mistral AI78.52026-04
Claude Sonnet 3.5Anthropic78.32026-04
Claude Sonnet 3.5 v2Anthropic78.32026-04
GPT-5 NanoOpenAI78.22026-04
Magistral SmallMistral AI78.22026-04
Magistral Small 2506Mistral AI78.22026-04
Gemma 3 27BGoogle DeepMind77.82026-04
Qwen2.5 32B InstructAlibaba / Qwen Team76.82026-04
Qwen2.5 32B Instruct AWQAlibaba / Qwen Team76.82026-04
GPT-4oOpenAI76.62026-04
GPT-4o (2024-05-13)OpenAI76.62026-04
GPT-4o (2024-08-06)OpenAI76.62026-04
GPT-4o (2024-11-20)OpenAI76.62026-04
Llama 4 Scout 17B 16EMeta76.52026-04
Llama 4 Scout 17B 16E InstructMeta76.52026-04
Llama-4-Scout-17B-16E-Instruct-FP8Meta76.52026-04
DeepSeek R1 Distill Qwen 7BDeepSeek75.32026-04
Qwen3 8BAlibaba / Qwen Team74.82026-04
Qwen3 8B AWQAlibaba / Qwen Team74.82026-04
Qwen3 8B BaseAlibaba / Qwen Team74.82026-04
Gemini 1.5 ProGoogle DeepMind74.32026-04
Llama 3.1 405BMeta73.82026-04
Llama 3.1 405B FP8Meta73.82026-04
Llama 3.1 405B InstructMeta73.82026-04
Llama 3.1 405B Instruct FP8Meta73.82026-04
Llama 3.3 70B Instruct NVFP4NVIDIA73.22026-04
Llama-3.3-70B-InstructMeta73.22026-04
Command ACohere72.52026-04
Command A ReasoningCohere72.52026-04
Phi 4 mini instructMicrosoft72.52026-04
GPT-4 TurboOpenAI72.22026-04
DeepSeek R1 Distill Llama 8BDeepSeek72.12026-04
Gemini 2.0 Flash LiteGoogle DeepMind72.12026-04
Grok 2xAI722026-04
Grok 2 (1212)xAI722026-04
Grok 2 LatestxAI722026-04
Grok 2 VisionxAI722026-04
Grok 2 Vision (1212)xAI722026-04
Grok 2 Vision LatestxAI722026-04
GPT-4.1 nanoOpenAI70.52026-04
Qwen2.5 Coder 32B InstructAlibaba / Qwen Team70.52026-04
Qwen2.5 Coder 32B Instruct AWQAlibaba / Qwen Team70.52026-04
GPT-4o miniOpenAI70.22026-04
Claude Haiku 3.5Anthropic69.52026-04
Claude Haiku 3.5 (latest)Anthropic69.52026-04
gemma 2 27B itGoogle DeepMind68.52026-04
Qwen2.5 14B InstructAlibaba / Qwen Team68.52026-04
Qwen2.5 14B Instruct AWQAlibaba / Qwen Team68.52026-04
Gemini 1.5 FlashGoogle DeepMind65.82026-04
Gemini 1.5 Flash-8BGoogle DeepMind65.82026-04
Gemma 3 12BGoogle DeepMind65.32026-04
Qwen3 4BAlibaba / Qwen Team65.32026-04
Qwen3 4B BaseAlibaba / Qwen Team65.32026-04
Qwen3 4B Instruct 2507Alibaba / Qwen Team65.32026-04
Qwen3 4B Instruct 2507 FP8Alibaba / Qwen Team65.32026-04
Codestral (latest)Mistral AI65.22026-04
Command R+Cohere65.22026-04
Llama 3.1 70BMeta64.52026-04
Llama 3.1 70B InstructMeta64.52026-04
Claude Opus 3Anthropic60.12026-04
Qwen2.5 7BAlibaba / Qwen Team58.52026-04
Qwen2.5 7B InstructAlibaba / Qwen Team58.52026-04
Qwen2.5 7B Instruct AWQAlibaba / Qwen Team58.52026-04
Yi 1.5 34B01.AI58.52026-04
Yi 1.5 34B Chat01.AI58.52026-04
Mixtral 8x22BMistral AI56.22026-04
Mixtral 8x22B Instruct v0.1Mistral AI56.22026-04
Command RCohere55.82026-04
Claude Sonnet 3Anthropic55.72026-04
Mistral NemoMistral AI55.32026-04
Mistral Nemo Base 2407Mistral AI55.32026-04
Mistral Nemo Instruct 2407Mistral AI55.32026-04
Phi 3.5 mini instructMicrosoft52.82026-04
gemma 2 9BGoogle DeepMind52.12026-04
gemma 2 9B itGoogle DeepMind52.12026-04
Claude Haiku 3Anthropic50.32026-04
Gemma 3 4BGoogle DeepMind48.52026-04

Showing the top 200 of 354.

Data

This page as JSON · Edit on GitHub