∞Bench's open-ended English QA split: answer a free-text question after reading a novel averaging around 193K tokens, scored by token-level F1 against the reference answer.
unassessed
| Category | long-context |
|---|---|
| Subcategory | long-document open-ended question answering over English novels |
| Page status | active |
| Metric | F1 (token-level, SQuAD-style, against the reference answer) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 351 |
| Dataset licence | MIT, per the OpenBMB/InfiniteBench GitHub repository; see the infinitebench family page. |
| Publisher | Department of Computer Science and Technology, Tsinghua University |
Given a full English novel (averaging about 192,600 tokens of context) and a question that requires aggregating scattered details (for example, a running total) or filtering out one specific fact among many similar candidates, the model produces a short free-text answer rather than choosing among options, as En.MC does. Questions come from the same human annotation pipeline as En.MC, and the reference prompt instructs the model to answer concisely, so the task combines long-range retrieval-and-reasoning with the separate demand of producing a short, precisely matched answer rather than a discursive one.
Open-ended, free-text question answering over a long document, with generation capped at a short number of output tokens; scored automatically against a reference answer rather than by an LLM judge.
No model card in ModelSpec reports this benchmark yet.