Embeds the 20 classic bAbI reasoning tasks as needles inside long PG19 book text, testing whether models can find and combine scattered facts at context lengths up to millions of tokens.
unassessed
| Category | long-context |
|---|---|
| Subcategory | long-context reasoning over bAbI tasks embedded in distractor text |
| Page status | active |
| Metric | accuracy (per-task answer match), typically averaged over QA1-QA5 as a headline figure |
| Direction | higher_is_better |
| Unit | % |
| Dataset licence | Apache-2.0 (code and PG-19 background text); BSD (underlying bAbI task source) |
BABILong tests whether a model can locate and combine facts scattered across an extremely long document, rather than concentrated near the question. It takes the 20 classic bAbI tasks -- synthetic reasoning problems originally designed to isolate single skills such as single-fact retrieval, two- or three-fact chaining, counting, listing, and simple induction or deduction -- and embeds each task's fact sentences inside long passages of unrelated book text drawn from the PG19 corpus, so the relevant sentences sit surrounded by large amounts of plausible-looking but irrelevant narrative. A model must find the scattered facts within that noise and answer a question that depends on combining them correctly. Because the amount of distractor text is controlled independently of the underlying reasoning task, BABILong isolates a model's ability to use a long context from its ability to solve the reasoning problem itself.
A long document built from PG19 book excerpts with bAbI fact sentences and a question inserted at controlled positions; free-form short-answer output, evaluated at fixed context lengths from 0k tokens up to 10M (extended to 50M in fine-tuned recurrent-memory experiments).
No model card in ModelSpec reports this benchmark yet.