503 hard multiple-choice questions with contexts from 8k to 2M words, built so a model must reason over long context rather than just retrieve, with human experts scoring only 53.7%.
unassessed
| Category | long-context |
|---|---|
| Subcategory | multiple-choice long-context deep understanding and reasoning (8k-2M words) |
| Page status | active |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 503 |
| Dataset licence | Apache-2.0 (per the Hugging Face dataset card) |
| Publisher | Tsinghua University and Zhipu.AI |
LongBench v2 gives a model a long context (8,000 to 2,000,000 words, mostly under 128K) and a four-option multiple-choice question that cannot be answered by simple retrieval. The 503 questions span six task categories: single-document QA, multi-document QA, long in-context learning, long-dialogue history understanding, code repository understanding, and long structured-data (table/graph) understanding. Questions were collected from nearly 100 highly educated contributors across diverse professional backgrounds, then filtered through automated and manual review for quality and genuine difficulty. The explicit design goal is to require deep understanding and multi-step reasoning over the full context rather than locating one planted fact, in contrast to needle-in-a-haystack-style long-context tests.
A long document, document set, dialogue history, code repository, or structured dataset plus a four-option multiple-choice question in; a single selected option (A-D) out. All questions carry a difficulty label (easy/hard) and a length bucket (short/medium/long), and the reference protocol reports both direct answering and answering with extended chain-of-thought reasoning.
No model card in ModelSpec reports this benchmark yet.