{
 "body": "\n## What it measures\n\nGPQA is a family of graduate-level, multiple-choice science question sets in biology, physics and\nchemistry, built specifically to resist being answered by internet search. Each question was written by\na PhD-level domain expert, then checked by other experts and by skilled non-expert annotators given open\nweb access and ample time. The family exists as three nested sets of\nincreasing selectivity \u2014 Extended, Main and Diamond \u2014 produced by progressively filtering down to the\nquestions that best separate expert understanding from lookup skill.\n\nThe \"Google-proof\" framing is the point: rather than reward retrieval of a fact a search engine can\nsurface directly, GPQA is filtered against that failure mode, so a high score is meant to reflect domain\nreasoning. All three subsets share the same four-option, single-turn, English-language, text-only format.\n\n## How it is scored\n\nEvery subset is scored on accuracy: the percentage of questions answered with the correct one of four\nletters. Random guessing scores 25%. Most current reporting uses zero-shot or few-shot chain-of-thought\nprompting, with OpenAI's simple-evals prompt (documented on gpqa_diamond.md) the closest thing to a\ncomparability standard; strict answer-parsing means a model that reasons correctly but formats its answer\nwrong can score below the random baseline. No single expert baseline covers the whole family: the\noriginal paper reports 65% (74% excluding self-identified mistakes) across its broader validation sample,\nwhile OpenAI's separately measured 69.7% PhD-expert baseline applies to Diamond specifically.\n\n## Dataset and licence\n\nThe authors started from a 564-question pool, held out 18 to leave the 546-question Extended set, then\nkept the 448 Extended questions where expert validators agreed to form Main. Diamond keeps the 198 Main\nquestions where both experts answered correctly\nand most non-expert validators did not. All three ship together under a CC BY 4.0 licence on Hugging Face\n(Idavidrein/gpqa), gated behind an access request there and password-protected on GitHub \u2014 deliberate\nfriction meant to slow the dataset's entry into future training corpora. A canary string is embedded for\nthe same reason. English only, text only, no train/test split within any subset.\n\n## Who publishes it\n\nGPQA was introduced by David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe\nPang, Julien Dirani, Julian Michael and Samuel R. Bowman, posted to arXiv in November 2023. The authors\nmaintain the reference dataset and code at github.com/idavidrein/gpqa. Independent trackers, notably\nEpoch AI, run and publish their own evaluations, though almost all current third-party tracking targets\nDiamond rather than Main or Extended.\n\n## Lineage\n\nGPQA names no formal predecessor. Within this repository, GPQA Diamond (`gpqa_diamond`) is the only\nsubset with its own page; it is the hardest of the three official subsets and by far the most widely\nreported today. Main and Extended do not yet have separate pages here. GPQA's own rapid saturation (see\nbelow) is one of the stated reasons newer, harder science and reasoning benchmarks exist, including\nOpenAI's FrontierScience and Scale AI/CAIS's Humanity's Last Exam, neither of which has a page in this\nrepository yet.\n\n## Saturation and contamination\n\nGPQA Diamond \u2014 the hardest and most tracked subset \u2014 moved from a genuinely hard benchmark to a largely\nsaturated one within two years: GPT-4 scored 39% at the November 2023 release, o1 reached 77.3% in\nSeptember 2024, and OpenAI reported GPT-5.2 at 92% in December 2025 (full detail on gpqa_diamond.md). No\nseparately sourced current top score for Main or Extended was found in this research, but because both\nare easier by construction than Diamond \u2014 Diamond is specifically the subset of Main where non-experts\nstruggled \u2014 models score at least as well on them, so the family is treated as saturated here without a\nseparate number for them. Contamination risk sits at medium, as with Diamond: the canary string and\ngated distribution slow leakage, but almost three years of circulation among a community that both\ntrains on and evaluates against these questions makes some leakage plausible.\n\n## How to run it\n\nlm-evaluation-harness's `gpqa` group tag runs all three subsets across five prompting variants each\n(zero-shot, n-shot, generative n-shot, chain-of-thought zero-shot, chain-of-thought n-shot). HELM's `gpqa`\nrun-spec function takes a `subset` argument and reads from the same Idavidrein/gpqa Hugging Face dataset\nfor any of the three. OpenCompass ships dataset-loading scaffolding for main, diamond and extended alike,\nbut its most commonly used config file enables only Diamond by default, with the other two present in\ncode but commented out. inspect_evals registers only `gpqa_diamond` \u2014 there is no family-wide\ninspect_evals task. So a bare \"GPQA\" score without a named subset should be treated as ambiguous.\n\n## Reading the numbers\n\nTreat an unqualified \"GPQA\" score as ambiguous until you know which of the three subsets it names: Main\nand Extended are measurably easier than Diamond. Diamond is the subset almost all current leaderboards\nand model announcements actually mean by \"GPQA,\" and it sits close to its ceiling (see gpqa_diamond.md\nfor current numbers). A high score on any\nsubset demonstrates graduate-level scientific recall and multi-step reasoning within a closed four-option\nformat; it says nothing about open-ended scientific work, which is why harder successors like\nFrontierScience and Humanity's Last Exam exist.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "gpqa",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Graduate-Level Google-Proof Q&A Benchmark"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "All three subsets were released together under a shared canary string and gated, password-protected distribution meant to slow leakage into training data. The licence bars public re-posting of examples. Even so, the dataset has circulated among researchers for almost three years and is widely used for both training and evaluation, so some leakage is plausible despite those measures.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC BY 4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 448,
   "size_note": "448 questions in the Main set. The family also ships a 546-question Extended set (Main's superset, after 18 questions were held out from an initial 564-question pool) and the 198-question Diamond subset (Main's hardest tier, kept where both experts answered correctly and most non-experts did not); see gpqa_diamond.md for the Diamond-specific page.\n",
   "splits": "single evaluation set per subset (main/diamond/extended), no train/test split",
   "url": "https://huggingface.co/datasets/Idavidrein/gpqa"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 2, slice A",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "gpqa",
   "inspect_evals": "",
   "lm_eval": "gpqa",
   "opencompass": "gpqa",
   "other": "lm-evaluation-harness's \"gpqa\" group tag runs all three subsets (main/diamond/extended) across five prompting variants each (zeroshot, n_shot, generative_n_shot, cot_zeroshot, cot_n_shot). HELM's \"gpqa\" run-spec function (get_gpqa_spec) takes a subset argument and reads from the same Idavidrein/gpqa dataset for any of the three. inspect_evals registers only gpqa_diamond, with no family-wide task. OpenCompass ships loader scaffolding for all three subsets, but its most commonly used config (gpqa_openai_simple_evals_gen_5aeece.py) enables only the Diamond split by default, with main and extended present in a gpqa_subsets dict but commented out.\n"
  },
  "id": "gpqa",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "gpqa_diamond"
   ]
  },
  "measures": "GPQA is a family of multiple-choice question sets in biology, physics and chemistry, written and validated by PhD-level domain experts specifically to resist answering by search. Every question was filtered so that expert validators agreed on the answer while skilled non-experts, given open web access and substantial time, mostly did not, so a high score is meant to reflect domain reasoning rather than lookup skill. The release ships as three overlapping sets of increasing selectivity: a 546-question Extended set, a 448-question Main set drawn from it, and the 198-question Diamond subset drawn from Main, which most current reporting treats as the default \"GPQA\" number.\n",
  "metric": {
   "baseline_note": "gpqa_diamond.md in this repository records OpenAI's PhD-expert baseline of 69.7% for the Diamond subset specifically. No separate expert baseline for the Main or Extended sets was found; the original paper reports 65% (74% excluding self-identified mistakes) for expert validators across its broader validation sample rather than for a single named subset.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 25,
   "unit": "%"
  },
  "name": "GPQA",
  "page_kind": "family",
  "paper": {
   "arxiv": "2311.12022",
   "title": "GPQA: A Graduate-Level Google-Proof Q&A Benchmark",
   "url": "https://arxiv.org/abs/2311.12022",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "David Rein",
    "Betty Li Hou",
    "Asa Cooper Stickland",
    "Jackson Petty",
    "Richard Yuanzhe Pang",
    "Julien Dirani",
    "Julian Michael",
    "Samuel R. Bowman"
   ],
   "org": "",
   "url": "https://github.com/idavidrein/gpqa"
  },
  "released": "2023-11",
  "repo_url": "https://github.com/idavidrein/gpqa",
  "saturation": {
   "as_of": "",
   "note": "No separately tracked current top score for the Main or Extended sets was found; most labs and trackers now report Diamond as the default GPQA number (see gpqa_diamond.md, which records a top score of 92% as of December 2025). Because Diamond is constructed as the hardest tier of Main, and Main is in turn the filtered core of Extended, models score at least as well on Main and Extended as on Diamond, so the family's easier subsets are treated as at least as saturated even without a separately sourced number for them.\n",
   "status": "saturated",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "GPQA: A Graduate-Level Google-Proof Q&A Benchmark",
    "url": "https://arxiv.org/abs/2311.12022"
   },
   {
    "accessed": "2026-09-08",
    "title": "idavidrein/gpqa (README, LICENSE, run_baseline.py)",
    "url": "https://github.com/idavidrein/gpqa"
   },
   {
    "accessed": "2026-09-08",
    "title": "Idavidrein/gpqa dataset card",
    "url": "https://huggingface.co/datasets/Idavidrein/gpqa"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness gpqa task group (main/diamond/extended x 5 prompt variants)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/gpqa"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals gpqa module (registers gpqa_diamond only)",
    "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/gpqa"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM capabilities_run_specs.py, get_gpqa_spec / run_spec_function(\"gpqa\")",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/run_specs/capabilities_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass gpqa dataset configs",
    "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/gpqa"
   }
  ],
  "status": "active",
  "subcategory": "graduate-level science Q&A",
  "summary": "Graduate-level multiple-choice science questions in biology, physics and chemistry, built to resist internet lookup; the family behind the widely-reported GPQA Diamond subset.",
  "tags": [
   "science",
   "multiple-choice",
   "phd-level",
   "chain-of-thought",
   "family-page"
  ],
  "task_format": "Four-option multiple-choice question in biology, physics or chemistry; the model returns a single letter answer, typically after chain-of-thought reasoning.\n"
 }
}