A bilingual English/Chinese suite of 21 tasks across six categories, testing long-context understanding at moderate lengths of roughly 5k-15k words per document.
unassessed
| Category | long-context |
|---|---|
| Subcategory | bilingual multitask long-context understanding (QA, summarization, few-shot learning, synthetic tasks, code completion) |
| Page status | active |
| Metric | task-specific metric (F1, ROUGE-L, classification accuracy, exact-match accuracy, or edit similarity), averaged across the 21 tasks for a headline score |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 4750 |
| Dataset licence | MIT for the LongBench code and packaging (GitHub repository); underlying source documents carry mixed original licences not restated in one place -- for example the paper describes MultiFieldQA's sources as arXiv papers, the ODC-BY-licensed C4 corpus, WuDaoCorpora, Chinese court judgment records, CC BY-SA Wikipedia, and Chinese government reports. |
| Publisher | Tsinghua University, Zhipu.AI, and the Institute of Automation, Chinese Academy of Sciences |
LongBench gives a model a document, or set of documents, roughly 5,000-15,000 words long and asks it to complete one of 21 tasks grouped into six categories: single-document QA, multi-document QA, summarization, few-shot in-context learning, synthetic retrieval/counting, and code completion. Fourteen tasks are in English, five are in Chinese, and two language-general code-completion tasks draw on GitHub repositories in Python, C#, and Java. English instances average 6,711 words; Chinese instances average 13,386 characters. Six of the 21 datasets are taken directly from earlier public benchmarks, ten are adapted and reprocessed for length, and five were built by the authors. A companion split, LongBench-E, resamples 13 of the 21 datasets for a length distribution balanced across 0-4k, 4-8k and 8k+ tokens, isolating length effects from task difficulty.
A long document or documents plus a short task-specific instruction (a question, a classification query, a summarization prompt, or a code-completion prefix) in; a short free-form answer, label, or code continuation out. Evaluated zero-shot by default, with per-task automatic metrics rather than a single shared scoring rule.
No model card in ModelSpec reports this benchmark yet.