A bilingual long-context QA benchmark with 11 datasets at five length levels from 16k to 256k words, built to fight knowledge leakage with confusing-fact insertion and keyword-recall metrics.
unassessed
| Category | long-context |
|---|---|
| Subcategory | bilingual long-context QA with confusion injection and keyword-recall scoring, five length levels to 256k words |
| Page status | active |
| Metric | keyword-recall-based F1 (most datasets), F1 with word blacklist (cmrc-mixup), or ROUGE-L with word blacklist (dureader-mixup); averaged per length level and across datasets |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 8645 |
| Dataset licence | MIT for the code and packaging (GitHub LICENSE file and the Hugging Face license tag); the GitHub repository also ships a separate LICENSE_CC file (CC BY-SA 4.0) without stating which of the two covers the released data itself versus the evaluation code. |
| Publisher | Tsinghua University; Infinigence-AI; Shanghai Jiao Tong University; The Chinese University of Hong Kong; Shanghai Artificial Intelligence Laboratory |
LV-Eval gives a model a long bilingual (English/Chinese) document, built by mixing real supporting passages with distracting ones, and asks a single-hop or multi-hop question over it. It spans 11 datasets -- six single-hop (including a fact-recall pressure test styled on needle-in-a-haystack) and five multi-hop, several adapted from sources such as LooGLE, HotpotQA-derived data, CMRC and DuReader -- each rendered at five length levels (16k, 32k, 64k, 128k, 256k words) using the same underlying question-answer pairs, so a model's degradation curve across lengths can be measured directly rather than compared across different questions. Most datasets additionally insert GPT-4-generated, human-revised "confusing facts" into the context and apply keyword-and-phrase replacement, so a model must reason from the given text rather than lean on memorized or common-sense knowledge, which the authors argue inflates scores on benchmarks built from unaltered public documents.
A long document (16k-256k words) mixing supporting and distracting passages, sometimes with inserted confusing facts, plus a single-hop or multi-hop question; a short free-form answer out, evaluated with greedy decoding. Ten of the 11 datasets are scored with a two-stage keyword-recall metric built from manually annotated answer keywords and a blacklist of non-informative words; the remaining two (cmrc-mixup, dureader-mixup) use a plain F1 or ROUGE-L variant with the same word blacklist.
No model card in ModelSpec reports this benchmark yet.