∞Bench's English multiple-choice split: pick the correct answer among four options after reading a novel averaging around 184K tokens, testing aggregation and filtering, not just retrieval.
unassessed
| Category | long-context |
|---|---|
| Subcategory | long-document multiple-choice question answering over English novels |
| Page status | active |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 229 |
| Dataset licence | MIT, per the OpenBMB/InfiniteBench GitHub repository; see the infinitebench family page. |
| Publisher | Department of Computer Science and Technology, Tsinghua University |
Given a full English novel (averaging about 184,400 tokens of context) and a question that requires locating and combining information spread across the book, the model picks one of four answer options; annotators were instructed to write challenging, plausible distractors. Questions follow the same annotation pipeline as En.QA, split into two reasoning styles: aggregation (compiling scattered details, such as a running total) and filtering (picking out one specific detail among many similar candidates, such as what a character wore at a particular point in the story). Because the answer is chosen from four given options rather than freely generated, En.MC isolates whether a model can find and combine the right long-range information without also being penalised for open-ended answer phrasing.
Four-option multiple-choice question answering over a long document; the model's free-text response is parsed for a single letter (A-D) and compared against the labelled option, functioning as an exact-match/accuracy score rather than a log-likelihood comparison.
No model card in ModelSpec reports this benchmark yet.