{
 "body": "\n## What it measures\n\nBoolQ tests whether a model can answer a naturally occurring yes/no question by reading a short passage and performing non-factoid, entailment-like inference rather than simple word matching. Each item pairs a question with a Wikipedia passage that answers it, and the model must output yes or no. Unlike constructed reading-comprehension datasets, the questions were not written by annotators looking at a passage: they were sampled from real, anonymised queries people had already typed into a search engine, then matched to a Wikipedia passage that answers them. The original authors argue this makes BoolQ questions harder and more natural than templated question-answer pairs, since nothing about how a question was phrased was shaped by the passage that would later answer it.\n\n## How it is scored\n\nScoring is plain accuracy against the gold yes/no label. The paper reports a majority-class baseline near 62% (most gold answers are \"yes\"), separate from the 50% two-class random-guess rate, and a human accuracy of about 90% measured on a sample. Its own best 2019 model, BERT pretrained and then transferred from the MultiNLI entailment dataset, reached 80.4%, and the paper's central finding is that transfer from entailment data helped more than transfer from paraphrase or extractive-QA data. Current harnesses score the same way: a generated or ranked yes/no answer compared against the gold label, with no partial credit.\n\n## Dataset and licence\n\nBoolQ totals 15,942 examples: 9,427 for training, 3,270 for validation, and 3,245 for test. Both the train and validation splits carry public labels; the test split's answers were never released by the original authors, and the Hugging Face mirror (`google/boolq`) exposes only the 12,697 labelled train-plus-validation rows as a result. The Hugging Face dataset card and the GitHub repository both give the licence as CC BY-SA 3.0. Passages come from Wikipedia articles; questions were filtered from real search queries down to ones answerable yes or no by a matched passage, then annotated and checked by crowd workers.\n\n## Who publishes it\n\nBoolQ comes from Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins and Kristina Toutanova at Google Research, published at NAACL 2019. The reference repository is `google-research-datasets/boolean-questions` on GitHub. BoolQ is also one of the eight tasks in the SuperGLUE benchmark suite, whose own leaderboard (`super.gluebenchmark.com`) is the closest thing to a maintained comparison point today, alongside the dataset's own repository, which notes a standalone leaderboard was planned but not confirmed live at the time of this research.\n\n## Lineage\n\nThis repository does not track a predecessor or successor for BoolQ directly. It is one of the eight SuperGLUE tasks, and this repository does not yet have a SuperGLUE family page for it to belong to. BoolQ Contrast Sets, a smaller set of human-perturbed items from Allen AI used by HELM as a robustness check, is a variant that also does not have its own page here.\n\n## Saturation and contamination\n\nThe paper's own numbers already point toward a close ceiling: human accuracy near 90%, a majority-class baseline near 62%, and an early BERT-based model already at 80.4% in 2019. This page could not read a current maintained leaderboard to confirm a specific present-day top score, but BoolQ is widely treated as saturated for frontier instruction-tuned models, consistent with its current role as one line inside broader suites rather than a standalone leaderboard. Contamination risk is high in practice: every harness reviewed for this page scores the labelled validation split, which has been public since 2019, rather than the genuinely held-out test split, so a model could plausibly have encountered both the questions and their answers during training.\n\n## How to run it\n\nlm-evaluation-harness implements BoolQ as task `boolq` inside its `super_glue` task group, scored by comparing the log-likelihood the model assigns to \"yes\" against \"no.\" HELM's `boolq` scenario adds an optional robustness check against the human-perturbed contrast sets. inspect_evals' `boolq` task pulls the Hugging Face validation split (3,270 samples) and computes simple accuracy. All three score the labelled validation split rather than the official held-out test split, so \"BoolQ test accuracy\" in most papers and model cards actually means validation accuracy.\n\n## Reading the numbers\n\nA high BoolQ score today mostly confirms a model can do basic passage-grounded yes/no inference, a capability frontier models cleared years ago, so it reads better as a floor check than as a differentiator between strong models. Because every mainstream harness scores the public validation split, treat scores from recent models with some caution for contamination rather than as clean evidence of reasoning ability. Compare a reported number against the roughly 90% human ceiling and roughly 62% majority baseline from the original paper: many current models exceed the ceiling that separated strong 2019 systems, which is more consistent with saturation and prior exposure than with genuine remaining headroom. Pair a BoolQ number with a harder, less exposed reading-comprehension or entailment benchmark before drawing conclusions about a model's inference ability from it alone.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "boolq",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Boolean Questions"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "The split every harness reviewed for this page actually scores -- the 3,270-example validation set -- has carried public answers since the 2019 release; only the 3,245-example official test split has ever had its answers withheld, and none of lm-evaluation-harness, HELM or inspect_evals evaluates against that held-out split. Passages are drawn from Wikipedia, a second route by which the source text, if not the exact question-answer pairing, could reach pretraining data.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC BY-SA 3.0 (Hugging Face dataset card and GitHub repository both state this)",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": 15942,
   "size_note": "15,942 examples total: 9,427 train, 3,270 validation (both labelled), and 3,245 test (labels never publicly released by the original authors). The Hugging Face mirror `google/boolq` exposes only the labelled train and validation splits -- 12,697 rows, confirmed via the datasets-server size endpoint -- because the true test split was never published with answers.\n",
   "splits": "train (9,427) / validation (3,270, labelled) / test (3,245, answers not publicly released)",
   "url": "https://huggingface.co/datasets/google/boolq"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 3, slice E"
  },
  "harness": {
   "bigbench": "",
   "helm": "boolq",
   "inspect_evals": "boolq",
   "lm_eval": "boolq",
   "opencompass": "",
   "other": ""
  },
  "id": "boolq",
  "last_updated": "",
  "leaderboard_url": "https://super.gluebenchmark.com/leaderboard",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "BoolQ tests whether a model can answer a naturally occurring yes/no question by reading a short passage and performing non-factoid, entailment-like inference rather than simple word matching. Each item pairs a question with a Wikipedia passage that answers it, and the model must output yes or no. Unlike constructed reading-comprehension datasets, the questions were not written by annotators looking at a passage: they were sampled from real, anonymised queries people had already typed into a search engine, then matched to a passage that answers them, which the original authors argue makes them harder and more natural than templated question-answer pairs.\n",
  "metric": {
   "baseline_note": "50% is the two-class random-guess rate. The paper separately reports a majority-class baseline of about 62% (the majority gold answer is \"yes\") and human accuracy of about 90% on a sample. Its own best model in 2019, BERT pretrained then transferred from MultiNLI, reached 80.4%.\n",
   "direction": "higher_is_better",
   "human_baseline": 90.0,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 50.0,
   "unit": "%"
  },
  "name": "BoolQ",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "1905.10044",
   "title": "BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions",
   "url": "https://arxiv.org/abs/1905.10044",
   "year": 2019
  },
  "publisher": {
   "authors": [
    "Christopher Clark",
    "Kenton Lee",
    "Ming-Wei Chang",
    "Tom Kwiatkowski",
    "Michael Collins",
    "Kristina Toutanova"
   ],
   "org": "Google Research",
   "url": "https://github.com/google-research-datasets/boolean-questions"
  },
  "released": "2019-05",
  "repo_url": "https://github.com/google-research-datasets/boolean-questions",
  "saturation": {
   "as_of": "",
   "note": "No specific current top score is recorded here: this page's attempts to read a maintained leaderboard did not succeed (the SuperGLUE leaderboard did not render as static content, and paperswithcode's historical BoolQ page no longer resolves to leaderboard data), so no number is claimed that was not read from a source. That said, the paper's own reference points already suggest a close ceiling -- human accuracy near 90%, majority-class baseline near 62%, and a BERT-based model at 80.4% in the year of release -- and BoolQ is treated in practice as saturated for instruction-tuned frontier models, which is why it now appears as one line inside broader suites (HELM, lm-evaluation-harness, inspect_evals) rather than as a standalone leaderboard.\n",
   "status": "saturated",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions",
    "url": "https://arxiv.org/abs/1905.10044"
   },
   {
    "accessed": "2026-09-08",
    "title": "google-research-datasets/boolean-questions GitHub repository",
    "url": "https://github.com/google-research-datasets/boolean-questions"
   },
   {
    "accessed": "2026-09-08",
    "title": "google/boolq dataset metadata, Hugging Face API",
    "url": "https://huggingface.co/api/datasets/google/boolq"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM boolq_scenario.py",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/boolq_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals boolq task",
    "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/boolq"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness super_glue tasks directory",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/super_glue"
   }
  ],
  "status": "saturated",
  "subcategory": "reading comprehension: naturally occurring yes/no questions requiring entailment-style inference over a passage",
  "summary": "15,942 naturally occurring yes/no reading-comprehension questions paired with a Wikipedia passage; part of SuperGLUE and now largely saturated for frontier models.",
  "tags": [
   "reading-comprehension",
   "yes-no",
   "entailment",
   "superglue",
   "saturated"
  ],
  "task_format": "Binary yes/no question answering given a short passage, zero- or few-shot, English."
 }
}