13,679 crowdsourced 4-way science questions; lm-eval scores log-likelihood accuracy on the 1,000-item test split after prepending the support paragraph.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | crowdsourced 4-way science exam questions (physics, chemistry, biology, earth science) |
| Page status | active |
| Metric | accuracy (acc and acc_norm) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 13679 |
| Dataset licence | CC-BY-NC-3.0 |
| Publisher | Allen Institute for AI |
SciQ is English multiple-choice science QA. Crowd workers read a textbook passage from CK-12 or OpenStax and wrote a question plus a correct answer and three distractors, with model-suggested distractors as hints. Subjects include physics, chemistry, biology, and earth science, from elementary through intro college. The paper's multiple-choice protocol is the question and four options; systems may retrieve background. EleutherAI lm-eval instead prepends the gold `support` field. That is a different and easier task than Table 2.
Four-way multiple choice. lm-eval task `sciq` uses output_type multiple_choice. doc_to_text is support (stripped) then Question and Answer. Choices are distractor1, distractor2, distractor3, correct_answer with doc_to_target index 3. Metrics acc and acc_norm. should_decontaminate true on support plus question. Default splits: train 11679, validation 1000, test 1000.
No model card in ModelSpec reports this benchmark yet.