21,208 multimodal science multiple-choice questions annotated with lectures and explanations for chain-of-thought training; the leaderboard has sat frozen well above the human baseline since early 2024.
unassessed
| Category | multimodal |
|---|---|
| Subcategory | multimodal science multiple-choice QA with lecture and explanation annotations |
| Page status | saturated |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 21208 |
| Dataset licence | CC BY-SA 4.0 (Hugging Face dataset card) |
| Publisher | University of California, Los Angeles (UCLA); Allen Institute for AI (AI2) -- multi-institution collaboration |
ScienceQA gives a model a science question, drawn from an elementary-through-high-school curriculum, together with optional context: an image (a diagram, photo or chart), a short hint, and always a multiple-choice answer set of two to five options that varies per question. Roughly half the questions include an image the question cannot be answered without; the rest are text-only. What sets ScienceQA apart from a plain multiple-choice quiz is that every question is also annotated with a "lecture" -- background knowledge relevant to the topic -- and a "solution," a worked explanation of the correct answer. The benchmark's own paper is not really about the multiple-choice task alone: it uses these lecture and solution annotations to train and evaluate models that generate a chain-of-thought explanation before answering, and shows this measurably improves accuracy for both few-shot GPT-3 and fine-tuned smaller models.
Multiple-choice question answering (two to five options, varying per question) over a science question that may include an accompanying image and a short hint; English only. Full evaluation runs the ~4,241-question test split, though many papers instead use a fixed, randomly-sampled 1,000-question "test-mini" subset for cost reasons.
No model card in ModelSpec reports this benchmark yet.