∞Bench tests long-context understanding on 12 synthetic and realistic tasks averaging around 200K tokens, well beyond the roughly 10K tokens most earlier long-context benchmarks used.
unassessed
| Category | long-context |
|---|---|
| Subcategory | synthetic and realistic long-document tasks beyond 100K tokens (retrieval, math, code, book QA, dialogue), English and Chinese |
| Page status | active |
| Metric | task-specific metric (accuracy, ROUGE-L-sum, or exact match, depending on task), averaged across the 12 tasks for a headline score |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 3946 |
| Dataset licence | MIT, per the OpenBMB/InfiniteBench GitHub repository; the paper itself and the Hugging Face dataset card do not separately state a licence. |
| Publisher | Department of Computer Science and Technology, Tsinghua University |
∞Bench (InfiniteBench) tests whether a model can use information spread across contexts far longer than most earlier long-context benchmarks tested, which the paper says averaged around 10K tokens. It combines 12 tasks in four groups: three synthetic retrieval tasks (finding a planted passkey, a number string, or a value in key-value pairs, buried in long noise text); two code tasks (spotting an injected bug across a repository, and simulating multi-step function execution); two math tasks (finding an extreme or median value in a long array, and tracking intermediate results through a long arithmetic expression); three novel/book comprehension tasks in English (summarization, aggregation-style question answering, and multiple-choice questions) plus one in Chinese; and one dialogue task that asks a model to identify a masked character's name from a long script. The synthetic tasks test raw long-range retrieval; the book and dialogue tasks test whether a model can reason over dependencies spread across a genuinely long document rather than just locate one planted fact.
The model is given a long document or synthetic long context (input lengths mostly in the 100K-200K token range) plus a task-specific question or instruction, and produces a free-form or multiple-choice answer, evaluated per task against that task's own metric.
No model card in ModelSpec reports this benchmark yet.