4,326 short, adversarially-collected fact-seeking questions with one indisputable answer, graded CORRECT/INCORRECT/NOT_ATTEMPTED by a model; GPT-4o scored under 40% at release.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | short-answer factuality, graded CORRECT / INCORRECT / NOT_ATTEMPTED by a model |
| Page status | active |
| Metric | grader-classified CORRECT / INCORRECT / NOT_ATTEMPTED, aggregated into accuracy, accuracy-given-attempted, and an F-score |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 4326 |
| Dataset licence | Not established for the question-and-answer data itself: the reference `openai/simple-evals` GitHub repository is MIT-licensed as code, and a third-party Hugging Face mirror (`basicv8vc/SimpleQA`) separately tags the dataset MIT, but neither is an authoritative statement of the data's own licence from OpenAI. |
| Publisher | OpenAI |
SimpleQA measures short-form factuality: whether a model can give a correct, brief answer to a fact-seeking question that has exactly one indisputable, unchanging answer, and, just as importantly, whether it recognises when it does not know the answer rather than guessing. Every question was hired-AI-trainer-written and adversarially selected against GPT-4o and GPT-3.5's own answers, then cross-checked by a second independent trainer and kept only when both agreed -- a construction OpenAI designed specifically to be challenging for frontier models at a point where earlier open-domain factual-recall sets such as TriviaQA had become saturated. This is a factuality-and-calibration benchmark, not a knowledge-breadth or reasoning test: a model can score well by attempting few questions but declining to guess on the rest, and OpenAI explicitly uses SimpleQA to study whether a model's stated confidence tracks its actual accuracy.
Short free-form answer generation to a single fact-seeking question, no supporting passage supplied; graded by a separate model classifier, not the model under test, into one of three categories rather than scored by string match against the reference answer.
No model card in ModelSpec reports this benchmark yet.