Tel Aviv University suite of seven long-text English tasks (summarisation, QA, NLI) that require synthesising information across naturally long documents.
unassessed
| Category | long-context |
|---|---|
| Subcategory | seven English long-document summarisation, QA, and NLI tasks in one text-to-text format |
| Page status | active |
| Metric | SCROLLS aggregate (geometric mean of per-task scores); task metrics ROUGE / F1 / EM |
| Direction | higher_is_better |
| Dataset size | 119495 |
| Dataset licence | MIT |
| Publisher | Tel Aviv University NLP (TAU NLP) |
SCROLLS tests whether a model can synthesise information over texts that are long in the wild, not padded sentences. Seven existing datasets are rewritten as one input-output string: GovReport and SummScreenFD summarisation, QMSum query-based meeting summarisation, Qasper and NarrativeQA question answering, QuALITY four-way long-document QA, and ContractNLI over NDAs. Queries are prepended to the document with a blank line. The intended skill is long-range reading, not short-passage overlap.
Unified sequence-to-sequence. Official scoring uses a hidden test set on the SCROLLS evaluator. lm-eval loads only train and validation from tau/scrolls zip JSONL (has_test_docs is false) and reformulates causal models with a Question/Answer prompt. QuALITY and ContractNLI are scored as log-likelihood multiple choice in lm-eval; the others generate until a newline.
No model card in ModelSpec reports this benchmark yet.