Multiple-choice QA over English passages averaging about 5,000 tokens; writers read the full article, and a hard subset beats speed-limited annotators.
unassessed
| Category | long-context |
|---|---|
| Subcategory | long-document English multiple-choice QA (~5k-token passages, four options) |
| Page status | active |
| Metric | accuracy (all); easy_acc and hard_acc also reported |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 6737 |
| Dataset licence | CC-BY-4.0 |
| Publisher | New York University (NYU-MLL) and collaborators |
QuALITY (Question Answering with Long Input Texts, Yes!) gives a model an English article of about 5,000 tokens and a four-option question written by a contributor who read the whole passage. Untimed validators must agree the item is answerable; a hard subset is the slice where speed-limited annotators mostly fail. The skill is long-document comprehension that skimming and lexical overlap do not solve. OpenCompass, the census harness for this id, runs the HTML-stripped development split as a generation multiple-choice task, not the hidden test set used by the NYU leaderboard.
Four-way multiple choice (A–D) over a long English article plus a question. Official scoring is accuracy on the held-out test set. OpenCompass is zero-shot generation with first_option_postprocess over ABCD.
No model card in ModelSpec reports this benchmark yet.