{
 "body": "\n## What it measures\n\nLongBench gives a model a document, or a small set of documents, roughly 5,000-15,000 words long and\nasks it to complete one of 21 tasks grouped into six categories: single-document QA, multi-document\nQA, summarization, few-shot in-context learning, synthetic retrieval and counting, and code\ncompletion. Fourteen tasks are in English, five are in Chinese, and two language-general\ncode-completion tasks (LCC, RepoBench-P) draw on GitHub repositories written in Python, C# and Java.\nEnglish instances average 6,711 words; Chinese instances average 13,386 characters. Six of the 21\ndatasets are taken directly from earlier public benchmarks such as HotpotQA and TriviaQA, ten are\nadapted from existing sources and reprocessed for length, and five -- including two of the three\nsynthetic tasks -- were built by the authors themselves.\n\nA companion split, LongBench-E, resamples 13 of the 21 datasets to give a length distribution\nbalanced across 0-4k, 4-8k and 8k+ tokens, which lets a user isolate how performance changes with\nlength independent of task difficulty.\n\n## How it is scored\n\nEach of the 21 datasets is scored with the metric suited to its task: F1 for extractive QA, ROUGE-L\nfor summarization and dialogue summarization, classification accuracy for the two few-shot\nclassification tasks (TREC and LSHT), exact-match accuracy for the three synthetic tasks, and edit\nsimilarity for the two code-completion tasks. A model's overall LongBench score averages across all\n21 datasets; category-level and per-task scores are also reported and carry most of the useful\nsignal, since the headline average blends several unrelated metrics. Text longer than a model's\ncontext window is truncated from the middle, preserving the beginning and end, following the \"Lost\nin the Middle\" finding that models attend less reliably to buried content. Zero-shot is the default\nreported setting.\n\n## Dataset and licence\n\n4,750 test instances span the 21 datasets: 19 hold 200 instances each except MultiFieldQA-en (150),\nand the two code-completion tasks hold 500 each. There is no separate train or validation split --\nLongBench is test-only. The GitHub repository (THUDM/LongBench, also mirrored on Hugging Face under\nthe zai-org namespace after Zhipu AI's Z.ai rebrand) is MIT-licensed, but the underlying documents\ncome from many original sources with their own terms: the paper describes MultiFieldQA's source\ndocuments alone as drawn from arXiv papers, the ODC-BY-licensed C4 corpus, WuDaoCorpora, Chinese\ncourt judgment records, CC BY-SA Wikipedia, and Chinese government reports, so no single licence\ncovers every item. All answers are public; the Hugging Face dataset card reports over 60,000\ndownloads.\n\n## Who publishes it\n\nYushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu,\nAohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang and Juanzi Li published LongBench in August 2023, with\nauthors from Tsinghua University, Zhipu.AI, and the Institute of Automation at the Chinese Academy of\nSciences; the paper later appeared at ACL 2024. The same Tsinghua/Zhipu.AI group actively maintains\nthe GitHub repository, most recently adding LongBench v2 in December 2024 and moving the original\nbenchmark's files into a `LongBench/` subdirectory of the same repo.\n\n## Lineage\n\nLongBench positions itself against earlier, narrower long-context suites (ZeroSCROLLS, L-Eval) that\nit says covered fewer task types, and against context-length probes that rely only on perplexity or\nsingle-fact retrieval. Its own successor, built by an overlapping author team, is LongBench v2\n(`longbenchv2`, December 2024): LongBench's fairly short 5k-15k-word contexts and largely extractive\ntasks stopped stressing newer, longer-context models, so LongBench v2 runs contexts from 8k to 2M\nwords and requires genuine multi-step reasoning rather than retrieval. This repository also\ncatalogues two unrelated long-context suites built around a similar goal of testing beyond a single\nshort document -- `infinitebench`, which pushes past 100K tokens with a different 12-task mix, and\n`ruler`, a synthetic length-controllable suite -- neither of which shares data or authorship with\nLongBench.\n\n## Saturation and contamination\n\nThe authors' own 2023 reference table shows real headroom at the time: the strongest model tested,\nChatGLM3-6B-32k, averaged 48.5% across English task categories and 52.8% across Chinese ones, with\nevery tested model under 30% on summarization in both languages. No current, actively maintained\nleaderboard with newer frontier models was found during this research; the project's attention and\nupdate cadence have shifted to LongBench v2 since its December 2024 release, so a present-day top\nscore for the original LongBench is not established here. Contamination risk is medium: six of the\n21 datasets are lifted directly from older public benchmarks with public answers, so a model trained\non those source datasets could see contamination independent of LongBench itself, while the synthetic\nand author-written subsets are less exposed; the full test set and its answers have been public and\ndownloadable without gating since August 2023.\n\n## How to run it\n\nlm-evaluation-harness implements all 21 tasks under task groups `longbench_single`, `longbench_multi`,\n`longbench_summarization`, `longbench_fewshot`, `longbench_synthetic` and `longbench_code`, each with\nan `_e` LongBench-E variant, alongside a `LongBench` tag for the standard 21-task suite. OpenCompass\ncarries an equivalent per-task config set under `longbench`. The original authors' own scripts\n(`pred.py`, `eval.py` in the GitHub repository) remain the reference implementation and additionally\nsupport retrieval- and summarization-based context-compression baselines alongside plain long-context\nevaluation. Because LongBench truncates from the middle on over-length inputs and several tasks use\nfree-form generation graded by automatic metrics rather than exact match, scores can shift with\nprompt template and generation-length settings even at a fixed task and language.\n\n## Reading the numbers\n\nA strong LongBench score shows a model can extract, summarize or manipulate information from a\nmoderately long (5k-15k word) bilingual document better than the 2023-era models the benchmark was\nbuilt to separate -- but the benchmark predates today's much longer context windows, and its\npredominantly extractive tasks are weaker evidence of deep multi-step reasoning over long context\nthan LongBench v2's harder multiple-choice format. Because the headline average blends F1, ROUGE-L,\nclassification accuracy, exact match and edit similarity across very different tasks, check the\ncategory and per-task breakdown rather than trusting the single number, and note whether a reported\nscore used the standard suite or the length-balanced LongBench-E split before comparing it against\nanother report.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "longbench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding"
  ],
  "category": "long-context",
  "contamination": {
   "note": "Six of the 21 datasets are lifted directly from older public benchmarks with public answers (including HotpotQA, TriviaQA and 2WikiMultihopQA), so a model trained on those source datasets could see contamination independent of LongBench itself. The four synthetic tasks and the author-written subsets are less exposed since they are newly constructed, but the full test set and its answers have been public and downloadable without gating since August 2023.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en",
    "zh"
   ],
   "license": "MIT for the LongBench code and packaging (GitHub repository); underlying source documents carry mixed original licences not restated in one place -- for example the paper describes MultiFieldQA's sources as arXiv papers, the ODC-BY-licensed C4 corpus, WuDaoCorpora, Chinese court judgment records, CC BY-SA Wikipedia, and Chinese government reports.",
   "modalities": [
    "text",
    "code"
   ],
   "public_test_set": true,
   "size": 4750,
   "size_note": "4,750 test instances across 21 datasets: 19 hold 200 instances each except MultiFieldQA-en (150), and the two code-completion tasks (LCC, RepoBench-P) hold 500 each. Test-only; there is no separate train or validation split. LongBench-E is a second, length-balanced resampling of 13 of the 21 datasets, not additional unique data.\n",
   "splits": "test only (4,750 instances across 21 datasets); LongBench-E is a separate length-balanced resampling of 13 datasets for the same test purpose.",
   "url": "https://huggingface.co/datasets/zai-org/LongBench"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 3, slice A",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "longbench (task groups longbench_single, longbench_multi, longbench_summarization, longbench_fewshot, longbench_synthetic, longbench_code; each has an _e LongBench-E variant, plus per-task tasks such as hotpotqa, gov_report, lcc)",
   "opencompass": "longbench (per-task configs, e.g. longbenchhotpotqa, longbenchgov_report, longbenchlcc, longbenchrepobench)",
   "other": "The authors' own pred.py/eval.py scripts in the THUDM/LongBench repository are the reference implementation and additionally support retrieval- and summarization-based context-compression baselines."
  },
  "id": "longbench",
  "last_updated": "2024-12",
  "leaderboard_url": "https://github.com/THUDM/LongBench/blob/main/LongBench/README.md",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [
    "longbenchv2"
   ],
   "variants": []
  },
  "measures": "LongBench gives a model a document, or set of documents, roughly 5,000-15,000 words long and asks it to complete one of 21 tasks grouped into six categories: single-document QA, multi-document QA, summarization, few-shot in-context learning, synthetic retrieval/counting, and code completion. Fourteen tasks are in English, five are in Chinese, and two language-general code-completion tasks draw on GitHub repositories in Python, C#, and Java. English instances average 6,711 words; Chinese instances average 13,386 characters. Six of the 21 datasets are taken directly from earlier public benchmarks, ten are adapted and reprocessed for length, and five were built by the authors. A companion split, LongBench-E, resamples 13 of the 21 datasets for a length distribution balanced across 0-4k, 4-8k and 8k+ tokens, isolating length effects from task difficulty.\n",
  "metric": {
   "baseline_note": "No single random or human baseline applies across all 21 tasks and five underlying metric types; the paper reports category-level and per-task scores per model instead of one baseline figure.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "task-specific metric (F1, ROUGE-L, classification accuracy, exact-match accuracy, or edit similarity), averaged across the 21 tasks for a headline score",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "LongBench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2308.14508",
   "title": "LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding",
   "url": "https://arxiv.org/abs/2308.14508",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Yushi Bai",
    "Xin Lv",
    "Jiajie Zhang",
    "Hongchang Lyu",
    "Jiankai Tang",
    "Zhidian Huang",
    "Zhengxiao Du",
    "Xiao Liu",
    "Aohan Zeng",
    "Lei Hou",
    "Yuxiao Dong",
    "Jie Tang",
    "Juanzi Li"
   ],
   "org": "Tsinghua University, Zhipu.AI, and the Institute of Automation, Chinese Academy of Sciences",
   "url": "https://github.com/THUDM/LongBench"
  },
  "released": "2023-08",
  "repo_url": "https://github.com/THUDM/LongBench",
  "saturation": {
   "as_of": "2023-08",
   "note": "The authors' own reference table (2023-era models only) shows real headroom: the strongest model tested, ChatGLM3-6B-32k, averaged 48.5% across English task categories and 52.8% across Chinese ones, with every model under 30% on summarization. No current, actively maintained leaderboard with newer frontier models was found during this research; the project's public attention and update cadence have shifted to LongBench v2 since its December 2024 release (its own README notes v1 files were moved into a `LongBench/` subdirectory), so a present-day top score for the original LongBench is not established here. A harder, actively tracked successor now exists, which is why this page marks the benchmark \"watch\" rather than \"open\".\n",
   "status": "watch",
   "top_score": 48.5
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Bai et al., arXiv:2308.14508)",
    "url": "https://arxiv.org/abs/2308.14508"
   },
   {
    "accessed": "2026-09-08",
    "title": "LongBench, full text (ar5iv)",
    "url": "https://ar5iv.labs.arxiv.org/html/2308.14508"
   },
   {
    "accessed": "2026-09-08",
    "title": "THUDM/LongBench GitHub repository (MIT licence, LongBench and LongBench v2 side by side)",
    "url": "https://github.com/THUDM/LongBench"
   },
   {
    "accessed": "2026-09-08",
    "title": "LongBench v1 subdirectory README (task table, leaderboard, evaluation instructions)",
    "url": "https://github.com/THUDM/LongBench/blob/main/LongBench/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "zai-org/LongBench dataset card, Hugging Face (formerly THUDM/LongBench)",
    "url": "https://huggingface.co/datasets/zai-org/LongBench"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness longbench task README (groups, tasks, changelog)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/longbench/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass longbench dataset configs",
    "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/longbench"
   }
  ],
  "status": "active",
  "subcategory": "bilingual multitask long-context understanding (QA, summarization, few-shot learning, synthetic tasks, code completion)",
  "summary": "A bilingual English/Chinese suite of 21 tasks across six categories, testing long-context understanding at moderate lengths of roughly 5k-15k words per document.",
  "tags": [
   "long-context",
   "bilingual",
   "multitask",
   "question-answering",
   "summarization",
   "code-completion",
   "synthetic",
   "zero-shot"
  ],
  "task_format": "A long document or documents plus a short task-specific instruction (a question, a classification query, a summarization prompt, or a code-completion prefix) in; a short free-form answer, label, or code continuation out. Evaluated zero-shot by default, with per-task automatic metrics rather than a single shared scoring rule.\n"
 }
}