15,942 naturally occurring yes/no reading-comprehension questions paired with a Wikipedia passage; part of SuperGLUE and now largely saturated for frontier models.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | reading comprehension: naturally occurring yes/no questions requiring entailment-style inference over a passage |
| Page status | saturated |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 15942 |
| Dataset licence | CC BY-SA 3.0 (Hugging Face dataset card and GitHub repository both state this) |
| Publisher | Google Research |
BoolQ tests whether a model can answer a naturally occurring yes/no question by reading a short passage and performing non-factoid, entailment-like inference rather than simple word matching. Each item pairs a question with a Wikipedia passage that answers it, and the model must output yes or no. Unlike constructed reading-comprehension datasets, the questions were not written by annotators looking at a passage: they were sampled from real, anonymised queries people had already typed into a search engine, then matched to a passage that answers them, which the original authors argue makes them harder and more natural than templated question-answer pairs.
Binary yes/no question answering given a short passage, zero- or few-shot, English.
No model card in ModelSpec reports this benchmark yet.