QASPER pairs 5,049 questions, written from only a title and abstract, with 1,585 full NLP papers whose text must supply the answer, making it a long-context benchmark by design.
unassessed
| Category | long-context |
|---|---|
| Subcategory | document-grounded question answering over full NLP research papers |
| Page status | active |
| Metric | Answer F1 (token overlap against reference answers, by answer type) and Evidence F1, per the reference implementation; harness scores vary (see How to run it) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 5049 |
| Dataset licence | CC-BY-4.0 |
| Publisher | Allen Institute for AI (AI2) |
QASPER tests whether a model can answer information-seeking questions by reading an entire research paper, not just its abstract. Each question was written by an NLP practitioner who saw only a paper's title and abstract -- deliberately withheld from the rest of the text -- and asked something they genuinely wanted to know about the full paper. A separate practitioner then answered the question after reading the whole document, supplying both the answer and the supporting evidence passages. Because the question-writer never saw the body of the paper, a system cannot answer well from the abstract alone the way it plausibly could on many other scientific-QA sets; the question is built to require the full document, which is what makes QASPER a long-context benchmark in practice even though it was not marketed primarily as one -- it is one of the tasks inside the SCROLLS long-document suite, described there simply as "question answering over research papers." Papers average many thousands of words, several times longer than a typical short-context QA passage, and the reference baseline model (Longformer Encoder Decoder, with a 16,384-token context window) was chosen specifically because ordinary short-context models cannot ingest a full paper at once.
Given a paper (or, depending on implementation, some portion of it) and a question, the model produces one of four answer types: a yes/no answer, a free-form text answer, a set of extractive spans copied from the paper, or "unanswerable." The reference evaluation also scores supporting evidence selection (which paragraphs justify the answer) alongside the answer itself.
No model card in ModelSpec reports this benchmark yet.