{
 "body": "\n## What it measures\n\n\u221eBench (InfiniteBench) tests whether a model can use information spread across contexts far longer than\nmost earlier long-context benchmarks tested -- the paper states those earlier suites averaged around 10K\ntokens, while this one averages roughly 200K. It combines 12 tasks in four groups. Three synthetic\nretrieval tasks bury a passkey, a number string, or a value from key-value pairs inside long noise text and\nask the model to find it. Two code tasks ask the model to spot an injected bug across a repository or\nsimulate multi-step function execution by hand. Two math tasks ask it to find an extreme or median value in\na long array, or track intermediate results through a long arithmetic expression. Four book/dialogue tasks\n-- three in English (summarization, aggregation-style question answering, and multiple-choice questions)\nand one in Chinese question answering, plus a separate English dialogue task that asks the model to name a\nmasked character from a long script -- test whether it can reason over dependencies spread across a\ngenuinely long document, rather than just locate one planted fact the way the synthetic tasks do.\n\n## How it is scored\n\nEach task uses its own metric: retrieval and math-find tasks and the multiple-choice book task use\naccuracy; code debugging uses multiple-choice accuracy; code execution and the number/key-value retrieval\ntasks use exact match; summarization uses ROUGE-L-Sum; and the long-arithmetic task scores the number of\ncorrect intermediate values produced before the first error. A model's headline \u221eBench number is typically\nthe unweighted average of its per-task scores across all 12 tasks, so it blends several different metrics\ninto one figure rather than measuring one consistent quantity.\n\n## Dataset and licence\n\n3,946 examples span the 12 tasks, ranging from 50 examples (Math.Calc) to 590 (Retrieve.PassKey and\nRetrieve.Number). Average context length across the suite is roughly 200K tokens; the retrieval tasks\naverage around 122K tokens of input, while the QA and summarization tasks run to roughly 185K-207K tokens.\nThe GitHub repository (OpenBMB/InfiniteBench) is MIT-licensed; neither the paper nor the Hugging Face\ndataset card states a separate licence for the data itself. The dataset is publicly downloadable without\ngating.\n\n## Who publishes it\n\n\u221eBench was introduced by Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu\nHan, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu and Maosong Sun, in the Department of Computer Science and\nTechnology at Tsinghua University, first posted to arXiv in February 2024. The authors maintain the\nreference data and evaluation code at github.com/OpenBMB/InfiniteBench, whose commit history shows no\nactivity since September 2024; no actively updated public leaderboard was found during this research pass.\n\n## Lineage\n\nNo predecessor, successor or formal variant of \u221eBench is catalogued in this repository. It is one of\nseveral long-context suites built around the same idea of testing far beyond typical context lengths;\nRULER is another such benchmark queued for this repository's benchmark census but not yet published here as\nof this page's writing, and readers comparing long-context numbers across sources should check which suite\n(and which task mix within it) produced a given score before treating two \"long-context\" numbers as the\nsame measurement.\n\n## Saturation and contamination\n\nAt release, GPT-4 led with an average of 45.63% across the 12 tasks (essentially perfect on\nRetrieve.PassKey, weaker elsewhere), ahead of Claude 2 (37.06%), Kimi-Chat (34.73%) and\nYaRN-Mistral-7B-128K (19.96%, 0% on Retrieve.KV) -- far below ceiling and clearly separating models at the\ntime. The reference repository has shown no commit activity since September 2024, and this pass found no\nindependently updated leaderboard with more recent scores, so current-day standing is not established\nhere; given how far frontier context windows have grown since 2024, today's top scores are plausibly well\nabove this reading. Contamination risk is medium: the four procedurally generated synthetic tasks resist\nverbatim memorization of the task format even though the specific published instances are fixed and\npublic, while the book- and dialogue-based tasks draw on real novels and scripts plausibly already present\nin pretraining corpora independent of this benchmark; no source quantifies an actual contamination rate.\n\n## How to run it\n\nThe authors' own evaluation scripts live under `src/` in the OpenBMB/InfiniteBench repository and call\nvarious model APIs per task. OpenCompass registers all 12 subtasks individually (for example\n`infinitebench_retrievepasskey`, `infinitebench_codedebug`, `infinitebench_zhqa`), the most likely route to\ncurrent scores under a shared harness. Not confirmed in the lm-evaluation-harness, HELM or BIG-bench lists.\n\n## Reading the numbers\n\nA high \u221eBench average shows a model can hold and use information across genuinely long contexts on a mix\nof retrieval and realistic tasks, not just pass a single \"needle in a haystack\" check -- it deliberately\nincludes tasks, like the book and math tasks, a model cannot solve by retrieval alone. Because the headline score averages several unrelated metrics (accuracy, ROUGE, exact match, an\nerror-position count) across tasks of very different difficulty, a similar overall number can hide very\ndifferent per-task profiles, so check the per-task breakdown rather than the average alone.\nGiven this page could not confirm a current leaderboard, treat any recent \u221eBench number with the\nunderstanding that it is being compared against a 2024 reference point rather than an actively tracked\nranking.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "infinitebench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "InfiniteBench",
   "\u221eBench"
  ],
  "category": "long-context",
  "contamination": {
   "note": "The dataset has been public without gating since February 2024. Its four synthetic tasks (the three retrieval tasks plus Math.Find and Math.Calc) are procedurally generated from random values, so the general task format is harder to answer from memorised training data than a fixed-answer benchmark, but the specific published instances are themselves fixed and could still be memorised verbatim. The book- and dialogue-based tasks draw on real novels and scripts that were plausibly already present in pretraining corpora independent of this benchmark. No source read for this page states a measured contamination rate.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en",
    "zh"
   ],
   "license": "MIT, per the OpenBMB/InfiniteBench GitHub repository; the paper itself and the Hugging Face dataset card do not separately state a licence.",
   "modalities": [
    "text",
    "code"
   ],
   "public_test_set": true,
   "size": 3946,
   "size_note": "3,946 examples across the 12 tasks: Retrieve.PassKey 590, Retrieve.Number 590, Retrieve.KV 500, En.Sum 103, En.QA 351, En.MC 229, Zh.QA 189, En.Dia 200, Code.Debug 394, Code.Run 400, Math.Calc 50, Math.Find 350. Average context length is roughly 200K tokens across the suite, with individual tasks such as Retrieve.PassKey and Retrieve.Number averaging around 122K tokens and the QA/summarization tasks running to roughly 185K-207K tokens.\n",
   "splits": "12 task-specific files, no shared train/test split; each task is its own fixed evaluation set",
   "url": "https://huggingface.co/datasets/xinrongzhang2022/InfiniteBench"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 2, slice F",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "infinitebench",
   "other": "The authors' own evaluation scripts live under `src/` in the OpenBMB/InfiniteBench repository, calling various model APIs per task. OpenCompass registers all 12 subtasks individually (for example `infinitebench_retrievepasskey`, `infinitebench_codedebug`, `infinitebench_zhqa`). Not confirmed in the lm-evaluation-harness, HELM or BIG-bench task lists.\n"
  },
  "id": "infinitebench",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "\u221eBench (InfiniteBench) tests whether a model can use information spread across contexts far longer than most earlier long-context benchmarks tested, which the paper says averaged around 10K tokens. It combines 12 tasks in four groups: three synthetic retrieval tasks (finding a planted passkey, a number string, or a value in key-value pairs, buried in long noise text); two code tasks (spotting an injected bug across a repository, and simulating multi-step function execution); two math tasks (finding an extreme or median value in a long array, and tracking intermediate results through a long arithmetic expression); three novel/book comprehension tasks in English (summarization, aggregation-style question answering, and multiple-choice questions) plus one in Chinese; and one dialogue task that asks a model to identify a masked character's name from a long script. The synthetic tasks test raw long-range retrieval; the book and dialogue tasks test whether a model can reason over dependencies spread across a genuinely long document rather than just locate one planted fact.\n",
  "metric": {
   "baseline_note": "No single random or human baseline applies across all 12 tasks, since they use different metrics and answer formats; the paper reports a per-task breakdown rather than one baseline number.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "task-specific metric (accuracy, ROUGE-L-sum, or exact match, depending on task), averaged across the 12 tasks for a headline score",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "\u221eBench (InfiniteBench)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2402.13718",
   "title": "\u221eBench: Extending Long Context Evaluation Beyond 100K Tokens",
   "url": "https://arxiv.org/abs/2402.13718",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Xinrong Zhang",
    "Yingfa Chen",
    "Shengding Hu",
    "Zihang Xu",
    "Junhao Chen",
    "Moo Khai Hao",
    "Xu Han",
    "Zhen Leng Thai",
    "Shuo Wang",
    "Zhiyuan Liu",
    "Maosong Sun"
   ],
   "org": "Department of Computer Science and Technology, Tsinghua University",
   "url": "https://github.com/OpenBMB/InfiniteBench"
  },
  "released": "2024-02",
  "repo_url": "https://github.com/OpenBMB/InfiniteBench",
  "saturation": {
   "as_of": "2024-02",
   "note": "At release, GPT-4 led with an average of 45.63% across the 12 tasks (100% on Retrieve.PassKey, weaker elsewhere), ahead of Claude 2 (37.06%), Kimi-Chat (34.73%) and YaRN-Mistral-7B-128K (19.96%, 0% on Retrieve.KV) -- far below ceiling and clearly separating models at the time. The OpenBMB/InfiniteBench repository's commit history shows no activity since September 2024, and this research pass found no independently updated public leaderboard with more recent scores, so current-day standing is not established here; given how far frontier context windows have grown since 2024, today's top scores are plausibly much higher than this 2024 reading.\n",
   "status": "unknown",
   "top_score": 45.63
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "\u221eBench: Extending Long Context Evaluation Beyond 100K Tokens",
    "url": "https://arxiv.org/abs/2402.13718"
   },
   {
    "accessed": "2026-09-08",
    "title": "\u221eBench, full text (ar5iv)",
    "url": "https://ar5iv.labs.arxiv.org/html/2402.13718"
   },
   {
    "accessed": "2026-09-08",
    "title": "xinrongzhang2022/InfiniteBench dataset card API, Hugging Face",
    "url": "https://huggingface.co/api/datasets/xinrongzhang2022/InfiniteBench"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenBMB/InfiniteBench repository",
    "url": "https://github.com/OpenBMB/InfiniteBench"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass infinitebench dataset configs",
    "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/infinitebench"
   }
  ],
  "status": "active",
  "subcategory": "synthetic and realistic long-document tasks beyond 100K tokens (retrieval, math, code, book QA, dialogue), English and Chinese",
  "summary": "\u221eBench tests long-context understanding on 12 synthetic and realistic tasks averaging around 200K tokens, well beyond the roughly 10K tokens most earlier long-context benchmarks used.",
  "tags": [
   "long-context",
   "retrieval",
   "code",
   "math",
   "multilingual",
   "synthetic"
  ],
  "task_format": "The model is given a long document or synthetic long context (input lengths mostly in the 100K-200K token range) plus a task-specific question or instruction, and produces a free-form or multiple-choice answer, evaluated per task against that task's own metric.\n"
 }
}