{
 "body": "\n## What it measures\n\nOpenBookQA measures multi-step reasoning over elementary science questions. Each question is multiple choice, and the dataset\u2019s additional configuration can provide a salient fact before the question. The intended skill combines that fact with common knowledge and language comprehension.\n\n## How it is scored\n\nOpenCompass uses an accuracy evaluator and extracts the first answer option among A, B, C, and D. Four choices imply a 25 percent uniform-choice baseline. Prompt templates differ between the main and fact-augmented configurations, so their scores should be reported separately.\n\n## Dataset and licence\n\nThe public dataset card reports 4,957 training, 500 validation, and 500 test examples for both main and additional configurations. It describes English text fields including question stem, choices, answer key, and, for additional data, `fact1`. The dataset card does not state a licence.\n\n## Who publishes it\n\nThe dataset was released by the Allen Institute for AI and introduced at EMNLP 2018 by Mihaylov, Clark, Khot, and Sabharwal. OpenCompass and lm-evaluation-harness provide integrations. No current authoritative leaderboard was established.\n\n## Lineage\n\nOpenBookQA is a standalone benchmark. This repository also contains `bangla_openbookqa`, a language variant, but it is a separate evaluation and should not be merged into the English score.\n\n## Saturation and contamination\n\nThe public 2018 test set creates high contamination risk. The benchmark remains useful for comparability, but scores can reflect memorization as well as reasoning. No authoritative current saturation measurement was found.\n\n## How to run it\n\nOpenCompass exposes `openbookqa` and `openbookqa_fact`. Both use zero-shot retrieval, generation, accuracy evaluation, and first-option postprocessing; the fact variant includes `fact1` in the prompt. The lm-evaluation-harness task is `openbookqa`; prompt details may differ.\n\n## Reading the numbers\n\nA high score indicates success on the benchmark\u2019s short science questions under its fixed choices. It does not establish broad scientific reasoning, robust open-book retrieval, or resistance to memorization. Record whether facts were supplied and which split and prompt implementation were used.\n\nThe fact-augmented configuration is especially sensitive to prompt wording: supplying the salient fact changes the information available to the model. Results should therefore identify the main or additional configuration rather than calling both simply OpenBookQA.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "openbookqa",
  "reasons": [
   "independently reviewed alias; retain canonical benchmark and protocol distinctions"
  ],
  "status": "alias",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "OBQA"
  ],
  "category": "knowledge",
  "contamination": {
   "note": "The 2018 questions and answer keys are public and old enough to plausibly occur in training data.",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 500,
   "size_note": "Main configuration has 4,957 train, 500 validation, and 500 test examples; the additional configuration has the same split counts.",
   "splits": "train, validation, test",
   "url": "https://huggingface.co/datasets/allenai/openbookqa"
  },
  "freshness": {
   "luna-batch-017": null,
   "luna-batch-017 (Codex coordinated)": null,
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna",
   "reviewed": "2026-09-08",
   "reviewed_by": "GPT-5.6 Luna independent review"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "openbookqa",
   "opencompass": "obqa",
   "other": ""
  },
  "id": "obqa",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "bangla_openbookqa"
   ]
  },
  "measures": "OpenBookQA asks grade-school science questions that require combining a supplied salient fact with common knowledge and language understanding. It is designed as an open-book exam style evaluation.",
  "metric": {
   "baseline_note": "",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 25,
   "unit": "percent"
  },
  "name": "OpenBookQA",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering",
   "url": "https://aclanthology.org/N18-1021/",
   "year": 2018
  },
  "publisher": {
   "authors": [
    "Todor Mihaylov",
    "Peter Clark",
    "Tushar Khot",
    "Ashish Sabharwal"
   ],
   "org": "Allen Institute for AI",
   "url": "https://allenai.org/data/open-book-qa"
  },
  "released": "2018",
  "repo_url": "https://huggingface.co/datasets/allenai/openbookqa",
  "saturation": {
   "as_of": "",
   "but no current authoritative top score was established.": null,
   "note": "The dataset is widely used and public",
   "status": "watch",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass OpenBookQA configuration",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/obqa/obqa_gen_9069e4.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "AllenAI OpenBookQA dataset card",
    "url": "https://huggingface.co/datasets/allenai/openbookqa"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenBookQA paper",
    "url": "https://aclanthology.org/N18-1021/"
   }
  ],
  "status": "active",
  "subcategory": "open-book question answering",
  "summary": "OpenBookQA tests multi-step science question answering with a small open book of facts and four answer choices.",
  "tags": [
   "science",
   "commonsense",
   "multiple-choice"
  ],
  "task_format": "Four-choice multiple-choice question, optionally accompanied by a fact; output A, B, C, or D."
 }
}