Graduate-level multiple-choice science questions in biology, physics and chemistry, built to resist internet lookup; the family behind the widely-reported GPQA Diamond subset.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | graduate-level science Q&A |
| Page status | active |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 448 |
| Dataset licence | CC BY 4.0 |
GPQA is a family of multiple-choice question sets in biology, physics and chemistry, written and validated by PhD-level domain experts specifically to resist answering by search. Every question was filtered so that expert validators agreed on the answer while skilled non-experts, given open web access and substantial time, mostly did not, so a high score is meant to reflect domain reasoning rather than lookup skill. The release ships as three overlapping sets of increasing selectivity: a 546-question Extended set, a 448-question Main set drawn from it, and the 198-question Diamond subset drawn from Main, which most current reporting treats as the default "GPQA" number.
Four-option multiple-choice question in biology, physics or chemistry; the model returns a single letter answer, typically after chain-of-thought reasoning.
No model card in ModelSpec reports this benchmark yet.