GPQA Diamond

198 PhD-written multiple-choice science questions built to resist lookup, the hardest subset of GPQA.

Also known as: GPQA-Diamond

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryreasoning
Subcategorygraduate-level science Q&A
Page statusactive
Metricaccuracy
Directionhigher_is_better
Unit%
Dataset size198
Dataset licenceCC BY 4.0

What it measures

GPQA Diamond tests whether a model can answer graduate-level multiple-choice questions in biology, physics and chemistry that are deliberately built to resist internet lookup. Each question was written and validated by a PhD-level domain expert, then filtered so that expert annotators agreed on the answer while most skilled non-experts, given open web access and significant time, did not. It is a single-turn, text-only, English-language task with four answer options and one correct letter, meant to reward domain reasoning rather than search skill.

Task format

Four-option multiple-choice question in biology, physics or chemistry; the model returns a single letter answer, typically after chain-of-thought reasoning.

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
Claude Mythos PreviewAnthropic94.552026-04
Gemini 3.1 Pro PreviewGoogle DeepMind94.32026-04
GPT-5.4OpenAI92.82026-04
Claude Opus 4.6Anthropic91.32026-04
o3-proOpenAI892026-04
o3OpenAI87.72026-04
o3-deep-researchOpenAI87.72026-04
GLM 5.1Z.ai (Zhipu AI)86.22026-04
o4-miniOpenAI82.32026-04
o4-mini-deep-researchOpenAI82.32026-04
Gemini 2.5 ProGoogle DeepMind80.52026-04
Gemini 2.5 Pro Preview 05-06Google DeepMind80.52026-04
Gemini 2.5 Pro Preview 06-05Google DeepMind80.52026-04
Gemini 2.5 Pro Preview TTSGoogle DeepMind80.52026-04
o1-proOpenAI80.52026-04
o3-miniOpenAI79.72026-04
GPT-5.1OpenAI79.12026-04
GPT-5.1 ChatOpenAI79.12026-04
GPT-5.1 CodexOpenAI79.12026-04
GPT-5.1 Codex MaxOpenAI79.12026-04
GPT-5.1 Codex miniOpenAI79.12026-04
Claude Opus 4Anthropic78.42026-04
GPT-5OpenAI77.22026-04
GPT-5 Chat (latest)OpenAI77.22026-04
GPT-5 ProOpenAI77.22026-04
GPT-5-CodexOpenAI77.22026-04
GPT-5.2OpenAI77.22026-04
GPT-5.2 ChatOpenAI77.22026-04
GPT-5.2 CodexOpenAI77.22026-04
GPT-5.2 ProOpenAI77.22026-04
GPT-5.3 Chat (latest)OpenAI77.22026-04
GPT-5.3 CodexOpenAI77.22026-04
GPT-5.3 Codex SparkOpenAI77.22026-04
GPT-5.4 miniOpenAI77.22026-04
GPT-5.4 nanoOpenAI77.22026-04
GPT-5.4 ProOpenAI77.22026-04
Claude Opus 4.5Anthropic77.02026-04
Claude Opus 4.5 (latest)Anthropic77.02026-04
Claude Opus 4.1Anthropic76.22026-04
Claude Opus 4.1 (latest)Anthropic76.22026-04
Grok 4xAI75.22026-04
Grok 4 FastxAI75.22026-04
Grok 4 Fast (Non-Reasoning)xAI75.22026-04
Grok 4.1 FastxAI75.22026-04
Grok 4.1 Fast (Non-Reasoning)xAI75.22026-04
Grok 4.20 (Non-Reasoning)xAI75.22026-04
Grok 4.20 (Reasoning)xAI75.22026-04
Grok 4.20 Multi-AgentxAI75.22026-04
Claude Sonnet 4Anthropic74.12026-04
Claude Sonnet 4.6Anthropic74.12026-04
Claude Opus 4 (latest)Anthropic73.82026-04
o1OpenAI73.32026-04
o1-previewOpenAI73.32026-04
DeepSeek R1 0528DeepSeek73.22026-04
DeepSeek R1 0528 NVFP4 v2NVIDIA73.22026-04
Claude Sonnet 4.5Anthropic72.52026-04
Claude Sonnet 4.5 (latest)Anthropic72.52026-04
DeepSeek R1DeepSeek71.52026-04
DeepSeek ReasonerDeepSeek71.52026-04
Qwen3 235B-A22BAlibaba / Qwen Team71.12026-04
Qwen3-Coder 480B-A35B InstructAlibaba / Qwen Team70.52026-04
Gemini 2.5 FlashGoogle DeepMind70.22026-04
Gemini 2.5 Flash ImageGoogle DeepMind70.22026-04
Gemini 2.5 Flash Image (Preview)Google DeepMind70.22026-04
Gemini 2.5 Flash LiteGoogle DeepMind70.22026-04
Gemini 2.5 Flash Lite Preview 06-17Google DeepMind70.22026-04
Gemini 2.5 Flash Lite Preview 09-25Google DeepMind70.22026-04
Gemini 2.5 Flash Preview 04-17Google DeepMind70.22026-04
Gemini 2.5 Flash Preview 05-20Google DeepMind70.22026-04
Gemini 2.5 Flash Preview 09-25Google DeepMind70.22026-04
Gemini 2.5 Flash Preview TTSGoogle DeepMind70.22026-04
GPT-4.1OpenAI69.52026-04
Claude Sonnet 4 (latest)Anthropic68.42026-04
GPT-5 MiniOpenAI68.22026-04
Claude Sonnet 3.7Anthropic68.02026-04
Gemma 4 31BGoogle DeepMind67.82026-04
gemma 4 31B itGoogle DeepMind67.82026-04
gemma 4 31B it GGUFUnsloth67.82026-04
Gemma 4 31B IT NVFP4NVIDIA67.82026-04
Gemma 4 26BGoogle DeepMind65.22026-04
gemma 4 26B A4B itGoogle DeepMind65.22026-04
gemma 4 26B A4B it GGUFUnsloth65.22026-04
Claude Sonnet 3.5Anthropic65.02026-04
Claude Sonnet 3.5 v2Anthropic65.02026-04
DeepSeek V3.2DeepSeek64.52026-04
DeepSeek V3.2 ExpDeepSeek64.52026-04
Grok 3xAI62.82026-04
Grok 3 FastxAI62.82026-04
Grok 3 Fast LatestxAI62.82026-04
Grok 3 LatestxAI62.82026-04
Llama 4 Maverick 17B 128E InstructMeta62.52026-04
Llama-4-Maverick-17B-128E-Instruct-FP8Meta62.52026-04
Qwen3 32BAlibaba / Qwen Team62.52026-04
Qwen3 32B AWQAlibaba / Qwen Team62.52026-04
Qwen3 32B NVFP4NVIDIA62.52026-04
DeepSeek V3.1DeepSeek62.32026-04
Claude Haiku 4.5Anthropic62.12026-04
Claude Haiku 4.5 (latest)Anthropic62.12026-04
Magistral Medium (latest)Mistral AI60.22026-04
Gemini 2.0 FlashGoogle DeepMind60.12026-04
GPT-4.1 miniOpenAI60.12026-04
o1-miniOpenAI60.02026-04
Claude Opus 3Anthropic59.42026-04
DeepSeek ChatDeepSeek59.12026-04
DeepSeek V2DeepSeek59.12026-04
DeepSeek V2 LiteDeepSeek59.12026-04
DeepSeek V2 Lite ChatDeepSeek59.12026-04
DeepSeek V3DeepSeek59.12026-04
DeepSeek V3 0324DeepSeek59.12026-04
QwQ PlusAlibaba / Qwen Team58.52026-04
Gemma 3 27BGoogle DeepMind58.22026-04
Gemini 1.5 ProGoogle DeepMind55.82026-04
Llama 4 Scout 17B 16EMeta55.82026-04
Llama 4 Scout 17B 16E InstructMeta55.82026-04
Llama-4-Scout-17B-16E-Instruct-FP8Meta55.82026-04
Mistral Large (latest)Mistral AI55.52026-04
Mistral Large 2.1Mistral AI55.52026-04
Mistral Large 3Mistral AI55.52026-04
Grok 3 MinixAI55.32026-04
Grok 3 Mini FastxAI55.32026-04
Grok 3 Mini Fast LatestxAI55.32026-04
Grok 3 Mini LatestxAI55.32026-04
DeepSeek R1 Distill Llama 70BDeepSeek55.22026-04
Qwen3 30B A3B Instruct 2507Alibaba / Qwen Team55.22026-04
Qwen3 30B A3B NVFP4NVIDIA55.22026-04
Qwen3 30B-A3BAlibaba / Qwen Team55.22026-04
GPT-5 NanoOpenAI55.12026-04
Qwen2.5 72B InstructAlibaba / Qwen Team54.82026-04
GPT-4oOpenAI53.62026-04
GPT-4o (2024-05-13)OpenAI53.62026-04
GPT-4o (2024-08-06)OpenAI53.62026-04
GPT-4o (2024-11-20)OpenAI53.62026-04
DeepSeek R1 Distill Qwen 32BDeepSeek52.82026-04
phi 4Microsoft52.52026-04
Phi 4 multimodal instructMicrosoft52.52026-04
Claude Haiku 3.5Anthropic51.22026-04
Claude Haiku 3.5 (latest)Anthropic51.22026-04
Magistral SmallMistral AI50.52026-04
Magistral Small 2506Mistral AI50.52026-04
Llama 3.3 70B Instruct NVFP4NVIDIA50.22026-04
Llama-3.3-70B-InstructMeta50.22026-04
Grok 2xAI50.12026-04
Grok 2 (1212)xAI50.12026-04
Grok 2 LatestxAI50.12026-04
Grok 2 VisionxAI50.12026-04
Grok 2 Vision (1212)xAI50.12026-04
Grok 2 Vision LatestxAI50.12026-04
Qwen3 14BAlibaba / Qwen Team50.12026-04
Qwen3 14B AWQAlibaba / Qwen Team50.12026-04
Qwen3 14B NVFP4NVIDIA50.12026-04
Llama 3.1 405BMeta49.82026-04
Llama 3.1 405B FP8Meta49.82026-04
Llama 3.1 405B InstructMeta49.82026-04
Llama 3.1 405B Instruct FP8Meta49.82026-04
GPT-4 TurboOpenAI49.32026-04
Gemini 2.0 Flash LiteGoogle DeepMind48.52026-04
Qwen2.5 32B InstructAlibaba / Qwen Team48.52026-04
Qwen2.5 32B Instruct AWQAlibaba / Qwen Team48.52026-04
GPT-4.1 nanoOpenAI48.32026-04
Command ACohere48.22026-04
Command A ReasoningCohere48.22026-04
gemma 2 27B itGoogle DeepMind48.12026-04
GPT-4o miniOpenAI46.12026-04
Claude Sonnet 3Anthropic45.52026-04
Gemma 3 12BGoogle DeepMind45.52026-04
DeepSeek R1 Distill Qwen 14BDeepSeek45.32026-04
Gemini 1.5 FlashGoogle DeepMind45.22026-04
Gemini 1.5 Flash-8BGoogle DeepMind45.22026-04
Codestral (latest)Mistral AI42.52026-04
Command R+Cohere42.52026-04
Llama 3.1 70BMeta42.52026-04
Llama 3.1 70B InstructMeta42.52026-04
Qwen2.5 Coder 32B InstructAlibaba / Qwen Team42.52026-04
Qwen2.5 Coder 32B Instruct AWQAlibaba / Qwen Team42.52026-04
Qwen3 8BAlibaba / Qwen Team42.52026-04
Qwen3 8B AWQAlibaba / Qwen Team42.52026-04
Qwen3 8B BaseAlibaba / Qwen Team42.52026-04
Phi 4 mini instructMicrosoft42.12026-04
Qwen2.5 14B InstructAlibaba / Qwen Team42.12026-04
Qwen2.5 14B Instruct AWQAlibaba / Qwen Team42.12026-04
Claude Haiku 3Anthropic41.22026-04
Meta Llama 3 70BMeta39.82024-07
GPT-4OpenAI39.72026-04
DeepSeek R1 Distill Qwen 7BDeepSeek38.52026-04
gemma 2 9BGoogle DeepMind38.22026-04
gemma 2 9B itGoogle DeepMind38.22026-04
Yi 1.5 9B01.AI37.92024-07
Yi 34B01.AI36.72024-07
Yi 1.5 34B01.AI36.52026-04
Yi 1.5 34B Chat01.AI36.52026-04
Yi 1.5 34B 32K01.AI36.32024-07
DeepSeek R1 Distill Llama 8BDeepSeek36.22026-04
Mixtral 8x22BMistral AI36.12026-04
Mixtral 8x22B Instruct v0.1Mistral AI36.12026-04
Yi 1.5 9B 32K01.AI35.92024-07
Yi 34B 200K01.AI35.72024-07
Infinity Instruct 3M 0625 Yi 1.5 9BBAAI35.42025-03
Command RCohere35.22026-04
Qwen2.5 7BAlibaba / Qwen Team35.22026-04
Qwen2.5 7B InstructAlibaba / Qwen Team35.22026-04

Showing the top 200 of 356.

Data

This page as JSON · Edit on GitHub