{
 "body": "\n## What it measures\n\nScienceQA gives a model a science question, drawn from an elementary-through-high-school curriculum, together with optional context: an image (a diagram, photo or chart), a short hint, and always a multiple-choice answer set of two to five options that varies per question. Roughly half the roughly 21,000 questions include an image the question cannot be answered without; the rest are text-only. What sets ScienceQA apart from a plain multiple-choice quiz is that every question is also annotated with a \"lecture\" -- background knowledge relevant to the topic -- and a \"solution,\" a worked explanation of the correct answer. The benchmark's own paper is not really about the multiple-choice task alone: it uses these lecture and solution annotations to train and evaluate models that generate a chain-of-thought explanation before answering, and shows this measurably improves accuracy for both few-shot GPT-3 and fine-tuned smaller models.\n\n## How it is scored\n\nScoring is plain accuracy against the single correct option. Because the number of options varies by question, there is no fixed random-guess rate; the paper instead measured an empirical \"Random Chance\" baseline of 39.83% by scoring uniform random guesses over each question's actual option set, and measured human performance the same way at 88.40% overall. Both figures are broken down identically by subject (natural/social/language science), context type (text, image, neither) and grade band (grades 1-6 versus 7-12) in the paper and on the official leaderboard; the paper's own baselines -- zero- and few-shot GPT-3, and fine-tuned UnifiedQA -- all scored well below the human figure at release.\n\n## Dataset and licence\n\nThe current Hugging Face mirror totals 21,208 examples -- 12,726 train, 4,241 validation, 4,241 test -- matching the paper's own \"~21k\" figure almost exactly, and is released under CC BY-SA 4.0. Every example carries an image field (empty for text-only questions), a question, a choice list, an answer index, an optional hint, the lecture/solution annotation pair described above, and grade, subject, topic, category and skill metadata. Because evaluating the full ~4,241-question test split is expensive against large or proprietary models, many papers instead report on a fixed, randomly-sampled 1,000-question \"test-mini\" subset; the official leaderboard tracks both.\n\n## Who publishes it\n\nScienceQA was introduced by Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark and Ashwin Kalyan, presented at NeurIPS 2022 under the title \"Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering.\" The lead author, Pan Lu, is at UCLA, and two co-authors, Oyvind Tafjord and Peter Clark, are AI2 researchers also behind this repository's `arc_challenge` page, making this a multi-institution collaboration rather than a single lab's release. The authors continue to maintain the reference GitHub repository and an interactive leaderboard at scienceqa.github.io/leaderboard.html.\n\n## Lineage\n\nNo predecessor or successor is tracked in this repository, but ScienceQA's lead author also co-authored MathVista (`mathvista`), a later multimodal reasoning benchmark in the same visual-chain-of-thought tradition, and ScienceQA is a natural point of comparison for this repository's other multimodal science and diagram pages, MMMU (`mmmu`) and AI2D (`ai2d`), even though none of the three formally descends from another. ScienceQA predates all three and is one of the earlier benchmarks to pair multimodal multiple-choice questions with explanation annotations specifically to support and measure chain-of-thought prompting.\n\n## Saturation and contamination\n\nThe official leaderboard -- read live for this page, and identical to the copy embedded in the GitHub README -- is topped by \"Multimodal-T-SciQ_Large\" at 96.18% overall (submitted May 2023), with the next four entries all above 93%, and dozens of its roughly 80 ranked entries sit above the 88.40% human baseline. The leaderboard's own maintainers date its last major update to December 2023 and its newest individual entry (KAM-CoT) to January 2024, meaning it has not moved in well over two years, and this repository's own model-card corpus contains zero mentions of \"scienceqa.\" The leaderboard carries its own caveat that entries are collected manually from papers rather than independently re-run, so exact rankings should be read with that in mind. Contamination risk is high: the full train, validation and test splits, including answers, lectures and solutions, have been public since 2022.\n\n## How to run it\n\nNo lm-evaluation-harness, inspect_evals, HELM or BIG-bench task name was confirmed for ScienceQA in the sources checked for this page. OpenCompass ships a `ScienceQA` config, but as currently written its reader configuration passes only the `question` and `choices` text fields into the prompt, with no `image` field referenced, and grades the free-form response with an LLM judge rather than the paper's original protocol -- meaning this specific harness path appears to exercise a text-only rendering of the benchmark rather than the image-grounded multimodal task ScienceQA is designed around. Vision-language-model evaluation toolkits more generally support the original image-plus-text task, though no specific task name for one was confirmed from a source opened for this page.\n\n## Reading the numbers\n\nA high ScienceQA score, on its own, now confirms very little: dozens of models on the official leaderboard already exceed the 88.40% human baseline, the leaderboard has been effectively frozen since early 2024, and no model card in this repository's own corpus currently reports it, all consistent with later multimodal suites (MMMU, MathVista) having displaced it for comparing frontier models. Before trusting any reported number, check whether it covers the full ~4,241-question test split or the smaller 1,000-question test-mini subset, and whether the harness that produced it actually supplied the question's image -- as the current OpenCompass configuration does not -- since a text-only score on an image-dependent question set is not comparable to the benchmark's original, image-grounded protocol.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "scienceqa",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "multimodal",
  "contamination": {
   "note": "The full train, validation and test splits, including answers, lectures and solutions, have been public since 2022. The leaderboard's saturation (see above) is itself consistent with heavy reuse of this fixed, fully public item set in both training and evaluation across the model families that populate it.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC BY-SA 4.0 (Hugging Face dataset card)",
   "modalities": [
    "text",
    "image"
   ],
   "public_test_set": true,
   "size": 21208,
   "size_note": "21,208 examples total -- 12,726 train, 4,241 validation, 4,241 test -- confirmed directly from the current Hugging Face mirror and matching the paper's own \"~21k\" figure almost exactly. Every example carries an image field (empty for text-only questions), question, choice list, answer index, optional hint, the lecture/solution annotation pair, and grade/subject/topic/category/ skill metadata.\n",
   "splits": "train (12,726) / validation (4,241) / test (4,241, answers public); many papers instead score a fixed 1,000-question random 'test-mini' subset",
   "url": "https://huggingface.co/datasets/derek-thomas/ScienceQA"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 3, slice B",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "ScienceQA (the current LLM-judge config's reader_cfg passes only the question and choices text fields, with no image field referenced, and grades free-form responses with an LLM judge rather than the paper's own protocol -- so as configured it appears to exercise a text-only rendering of the task, not the image-grounded multimodal one)",
   "other": ""
  },
  "id": "scienceqa",
  "last_updated": "2024-01",
  "leaderboard_url": "https://scienceqa.github.io/leaderboard.html",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "ScienceQA gives a model a science question, drawn from an elementary-through-high-school curriculum, together with optional context: an image (a diagram, photo or chart), a short hint, and always a multiple-choice answer set of two to five options that varies per question. Roughly half the questions include an image the question cannot be answered without; the rest are text-only. What sets ScienceQA apart from a plain multiple-choice quiz is that every question is also annotated with a \"lecture\" -- background knowledge relevant to the topic -- and a \"solution,\" a worked explanation of the correct answer. The benchmark's own paper is not really about the multiple-choice task alone: it uses these lecture and solution annotations to train and evaluate models that generate a chain-of-thought explanation before answering, and shows this measurably improves accuracy for both few-shot GPT-3 and fine-tuned smaller models.\n",
  "metric": {
   "baseline_note": "Because the number of options varies by question, there is no fixed random-guess rate; the paper instead measured an empirical \"Random Chance\" baseline of 39.83% by scoring uniform random guesses over each question's actual option set, and measured human performance the same way at 88.40% overall. Both are broken down identically by subject (natural/social/language science), context type (text, image, neither) and grade band (grades 1-6 versus 7-12) in the paper and on the official leaderboard; the paper's own baselines (zero- and few-shot GPT-3, fine-tuned UnifiedQA) all scored well below the human figure at release.\n",
   "direction": "higher_is_better",
   "human_baseline": 88.4,
   "max_score": 100.0,
   "name": "accuracy",
   "random_baseline": 39.83,
   "unit": "%"
  },
  "name": "ScienceQA",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2209.09513",
   "title": "Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering",
   "url": "https://arxiv.org/abs/2209.09513",
   "year": 2022
  },
  "publisher": {
   "authors": [
    "Pan Lu",
    "Swaroop Mishra",
    "Tony Xia",
    "Liang Qiu",
    "Kai-Wei Chang",
    "Song-Chun Zhu",
    "Oyvind Tafjord",
    "Peter Clark",
    "Ashwin Kalyan"
   ],
   "org": "University of California, Los Angeles (UCLA); Allen Institute for AI (AI2) -- multi-institution collaboration",
   "url": "https://scienceqa.github.io"
  },
  "released": "2022-09",
  "repo_url": "https://github.com/lupantech/ScienceQA",
  "saturation": {
   "as_of": "2024-01",
   "note": "The official leaderboard -- read live for this page, and identical to the copy embedded in the GitHub README -- is topped by \"Multimodal-T-SciQ_Large\" at 96.18% overall (submitted May 2023), with the next four entries all above 93%, and dozens of its roughly 80 ranked entries sit above the 88.40% human baseline. The leaderboard's own maintainers date its last major update to December 2023 and its newest individual entry (KAM-CoT) to January 2024, meaning it has not moved in well over two years; this repository's own model-card corpus contains zero mentions of \"scienceqa.\" The leaderboard carries its own caveat that entries are collected manually from papers rather than independently re-run, so exact rankings should be read with that in mind.\n",
   "status": "saturated",
   "top_score": 96.18
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering (arXiv abstract)",
    "url": "https://arxiv.org/abs/2209.09513"
   },
   {
    "accessed": "2026-09-08",
    "title": "lupantech/ScienceQA GitHub repository (README, embedded leaderboard table, NeurIPS 2022 citation)",
    "url": "https://github.com/lupantech/ScienceQA"
   },
   {
    "accessed": "2026-09-08",
    "title": "ScienceQA official interactive leaderboard",
    "url": "https://scienceqa.github.io/leaderboard.html"
   },
   {
    "accessed": "2026-09-08",
    "title": "derek-thomas/ScienceQA dataset card, Hugging Face (license, split sizes, feature schema)",
    "url": "https://huggingface.co/datasets/derek-thomas/ScienceQA"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass ScienceQA LLM-judge dataset config",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/ScienceQA/ScienceQA_llmjudge_gen_f00302.py"
   }
  ],
  "status": "saturated",
  "subcategory": "multimodal science multiple-choice QA with lecture and explanation annotations",
  "summary": "21,208 multimodal science multiple-choice questions annotated with lectures and explanations for chain-of-thought training; the leaderboard has sat frozen well above the human baseline since early 2024.",
  "tags": [
   "multimodal",
   "science-qa",
   "chain-of-thought",
   "multiple-choice",
   "saturated"
  ],
  "task_format": "Multiple-choice question answering (two to five options, varying per question) over a science question that may include an accompanying image and a short hint; English only. Full evaluation runs the ~4,241-question test split, though many papers instead use a fixed, randomly-sampled 1,000-question \"test-mini\" subset for cost reasons.\n"
 }
}