Free-form questions about entire books and movie scripts, scored against human reference answers, though most harnesses answer from a summary rather than the full narrative.
unassessed
| Category | long-context |
|---|---|
| Subcategory | generative question answering over long narratives (books and screenplays) |
| Page status | active |
| Metric | Rouge-L, Bleu-1, Bleu-4 and Meteor against two reference answers (the paper's own metric suite); HELM instead reports token-level F1 as its main metric, and OpenCompass scores with a TriviaQA-style evaluator |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 46765 |
| Dataset licence | Apache-2.0 |
| Publisher | DeepMind |
NarrativeQA gives a model a question about a book or a movie script and asks it to produce a free-form answer, the way a reader who has actually read the work could. The stories come from Project Gutenberg (books) and sites such as IMSDb (movie scripts); each was paired with its Wikipedia plot summary, and Amazon Mechanical Turk annotators wrote ten question-answer pairs per story after reading only the summary, not the full text, so that questions probe the underlying narrative rather than a specific sentence. The paper defines two settings that differ enormously in difficulty: "summaries only," where the model reads the same Wikipedia summary the annotators used, and "stories only," where it must find the answer somewhere in the full book or script, which can run to tens of thousands of words. Only the second setting exercises the long-document comprehension the benchmark is best known for.
Free-form text generation: given a passage (a summary or a full story, depending on setting) and a question, produce a short answer. Each question has two independently written human reference answers. The official evaluation script scores against both metrics from the paper (Bleu-1, Bleu-4, Meteor and Rouge-L); HELM's implementation instead reports a single token-level F1 as its main metric, which does not appear in the original paper at all.
No model card in ModelSpec reports this benchmark yet.