A last-word-prediction test built from narrative passages that humans can only complete correctly after reading the whole passage, not just the final sentence.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | broad-context word prediction |
| Page status | active |
| Metric | accuracy (next-word exact match); perplexity also reported |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 5153 |
| Dataset licence | CC BY 4.0 |
| Publisher | University of Trento (CIMeC); University of Amsterdam |
LAMBADA gives a model a narrative passage -- several sentences of context plus one final "target" sentence with its last word removed -- and asks it to predict that missing word. Passages were filtered so human readers shown only the final sentence could not guess the word, while readers shown the whole passage could. Succeeding therefore requires tracking information across the broader discourse -- who is speaking, what was named earlier, what the scene is -- rather than relying on local, sentence-level statistics, which is why the original paper frames it as testing "genuine understanding of broad context."
Language modelling / cloze: passage with its final word withheld in, a single predicted word out, scored as next-token prediction under a causal language model rather than multiple choice.
No model card in ModelSpec reports this benchmark yet.