A 2,780-question yes/no benchmark whose reasoning steps are never stated in the question, testing whether a model can infer and chain an implicit strategy to reach the answer.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | implicit multi-hop strategy question answering |
| Page status | active |
| Metric | accuracy (exact match on the yes/no answer) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 2780 |
| Dataset licence | MIT |
| Publisher | Allen Institute for AI |
StrategyQA gives a model an open-domain yes/no question, such as "Did Aristotle use a laptop?", where the steps needed to answer it are never spelled out and must be inferred. Answering requires the solver to work out an implicit strategy (for the laptop example, that Aristotle died centuries before the invention of laptops), then chain several individually easy lookups into a final boolean answer. Each question was crowdsourced together with a human-written decomposition into reasoning steps and a set of supporting Wikipedia evidence paragraphs, so the dataset also supports scoring intermediate retrieval and decomposition quality, not only the final yes or no.
Open-domain yes/no question in; boolean answer out. The original release also pairs each question with a step-by-step decomposition and Wikipedia evidence paragraphs, which some evaluation setups use to score retrieval or reasoning-chain quality rather than only the final answer.
No model card in ModelSpec reports this benchmark yet.