20 synthetic reading-comprehension toy tasks, from single-fact retrieval to induction and path-finding, that check basic reasoning skills; long saturated, now mainly the substrate for BABILong.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | 20 synthetic single- and multi-fact reading-comprehension reasoning toy tasks |
| Page status | saturated |
| Metric | accuracy (exact or quasi-exact match) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 40000 |
| Dataset licence | CC BY 3.0, per the Hugging Face mirror's dataset card; no separate licence file was confirmed directly from the original repository during this research |
| Publisher | Facebook AI Research |
bAbI gives a model a short, auto-generated story -- a sequence of simple sentences describing characters moving between locations, holding or giving objects, and so on -- followed by a question that requires combining one or more of those sentences to answer. It is not one task but 20, each isolating a distinct skill: single- and multi-fact retrieval, counting, listing, negation, indefinite knowledge, basic and compound coreference, conjunctions, positional and size reasoning, path-finding, deduction, induction, and inferring motivation. The authors designed it explicitly as a set of prerequisite toy tasks rather than a difficulty benchmark in itself: a system that fails a given task, they argue, is missing a specific skill needed for more general language understanding and dialogue, and the tasks exist to let researchers classify and target such failures rather than to differentiate frontier models.
"Passage: <story> Question: <question> Answer: <answer>" -- the model reads an ordered list of short fact sentences (a "story") and answers a question about it with a single word or short phrase (a name, a location, a count, a yes/no, or, for the one path-finding task, a short direction sequence). There are no answer options; scoring is free-form generation.
No model card in ModelSpec reports this benchmark yet.