{
 "body": "\n## What it measures\n\nLV-Eval gives a model a long bilingual (English or Chinese) document, assembled by mixing real\nsupporting passages with distracting ones, and asks a single-hop or multi-hop question over it. It\nspans 11 datasets -- six single-hop, including a fact-recall pressure test styled on\nneedle-in-a-haystack, and five multi-hop -- several adapted from earlier sources such as LooGLE,\nHotpotQA-derived data, CMRC and DuReader. Each of the 11 datasets is rendered at five length levels\n(16k, 32k, 64k, 128k and 256k words) using the same underlying question-answer pairs, so a model's\ndegradation curve across lengths can be measured directly rather than inferred by comparing\ndifferent questions at different lengths.\n\nMost datasets also insert GPT-4-generated, human-revised \"confusing facts\" into the context and\napply keyword-and-phrase replacement to both the context and the reference answers. Both techniques\nare meant to force a model to reason from the given text rather than lean on memorized or\ncommon-sense knowledge, which the authors argue inflates scores on long-context benchmarks built\nfrom unaltered public documents.\n\n## How it is scored\n\nTen of the 11 datasets use a two-stage, keyword-recall-based metric: manually annotated \"answer\nkeywords\" (the critical words or phrases in the reference answer) must be recalled above a\nthreshold before a second stage removes blacklisted, non-informative words and computes F1 against\nthe remaining text. The two exceptions, cmrc-mixup and dureader-mixup, use a plain F1 or ROUGE-L\nscore with the same word blacklist but no keyword-recall gate, because their reference answers are\nalready concise or already long-form respectively. There is no single fixed random or human\nbaseline; the benchmark is built for controlled comparison of a model against itself across the\nfive length levels and against other evaluated models, not against an absolute ceiling.\n\n## Dataset and licence\n\nThe 11 datasets share 1,331 unique question-answer pairs (596 single-hop, 735 multi-hop), each\nrendered as a graded context at all five length levels, for 8,645 total test instances by this\npage's count of the authors' published per-dataset context totals. The two fact-recall datasets\nhold just one QA pair each, repeated across 200 differently arranged contexts per length level as a\ndedicated pressure test. The GitHub repository is MIT-licensed for its code, matching the Hugging\nFace dataset card's license tag, but the repository also carries a separate LICENSE_CC file (CC\nBY-SA 4.0) without stating which licence governs the released data versus the evaluation scripts.\nThe Hugging Face card's own description text claims \"12 finegrained tasks,\" but only 11 are named\nin the paper, the README's task tables and the repository's ZIP files -- this page could not locate\na 12th task and treats that count as an error in the card.\n\n## Who publishes it\n\nTao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao,\nDahua Lin, Boxun Li, Guohao Dai, Shengen Yan and Yu Wang published LV-Eval in February 2024, with\naffiliations spanning Tsinghua University, Infinigence-AI, Shanghai Jiao Tong University, The\nChinese University of Hong Kong and Shanghai Artificial Intelligence Laboratory. Infinigence-AI\nhosts the reference repository and the Hugging Face dataset. The GitHub repository has not been\npushed to since August 2024, but the paper text itself was revised as recently as October 2025\n(arXiv v3) to report results on a wider set of newer models, without an accompanying code or\nreference-table update.\n\n## Lineage\n\nLV-Eval has no formal predecessor or successor tracked in this repository. Its stated motivation is\na gap in earlier long-context benchmarks, which the authors say average only 5k-21k words of context\nand are vulnerable to \"knowledge leakage\" from unaltered, previously public source documents. This\nrepository separately catalogues several other long-context suites built around related but distinct\nideas -- `longbench` and `longbenchv2` (bilingual and English long-context suites from a different,\nTsinghua/Zhipu.AI-led team), `infinitebench` (English/Chinese, pushing past 100K tokens with a\ndifferent 12-task mix) and `ruler` (NVIDIA's length-controllable synthetic suite) -- none of which\nshares data, authorship or its specific confusing-fact-insertion technique with LV-Eval.\n\n## Saturation and contamination\n\nLV-Eval is not saturated. The originally published 2024 reference table shows every evaluated\n7B-scale model well below 50% even at the easiest 16k level, falling into single digits by 256k. The\npaper's October 2025 text revision reports a wider 15-model sweep adding Moonshot-v1-128k,\nQwen2.5-72B-Instruct and Llama-3.1-70B-Instruct, stating these newer, larger models \"obtain average\nscores exceeding 40 at 16k and 32k lengths\" -- still well short of a ceiling, though this page could\nnot locate an exact figure for that run beyond this qualitative statement. Contamination risk is\nassessed as low: the benchmark's confusing-fact insertion and keyword replacement are specifically\ndesigned to blunt reliance on memorized or common-sense knowledge, though several of its source\ndatasets are older public benchmarks in their own right, and the full LV-Eval test set has been\npublic since February 2024 with no held-out portion.\n\n## How to run it\n\nOpenCompass ships all 11 datasets as separate per-dataset configs under\n`opencompass/configs/datasets/lveval/` (for example `lvevalhotpotwikiqa_mixup`,\n`lvevalfactrecall_en`), aggregated into a single `LVEval_datasets` list; no lm-evaluation-harness,\ninspect_evals or HELM implementation was found during this research. The authors' own\n`prediction.py` and `evaluation.py` scripts in infinigence/LVEval remain the reference\nimplementation, with batch shell scripts for data-parallel local evaluation and a separate script\nfor API-based commercial models such as GPT-4. Because the keyword-recall metric depends on manually\nannotated answer keywords and a word blacklist rather than plain string overlap, and because model\ncontext windows shorter than the target length force middle-truncation, reported scores can be\nsensitive to how a given harness handles truncation and answer-keyword matching even at the same\nnominal length.\n\n## Reading the numbers\n\nA strong LV-Eval score, especially one that holds up at 128k-256k rather than only at 16k-32k, is\ncomparatively good evidence a model can actually use a long context under adversarial conditions --\ndistractor passages, inserted confusing facts and knowledge-leakage mitigations all working against\nsimple retrieval or memorization. Because scores are reported per length level and blend a\nkeyword-recall metric that differs from plain F1 or ROUGE-L, always check which length level and\nwhich underlying metric produced a given number before comparing it across models or against another\nlong-context benchmark. A model's degradation curve across the five lengths, not just its score at\none length, is the more informative signal LV-Eval is built to expose.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "lveval",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "LVEval",
   "LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K"
  ],
  "category": "long-context",
  "contamination": {
   "note": "LV-Eval was built specifically to reduce two contamination-adjacent problems the authors call \"knowledge leakage\": documents mixing in confusing, GPT-4-generated facts and keyword/phrase replacement in both context and answers are meant to force reliance on the given text rather than on facts a model may already know from pretraining. That said, several source datasets (LooGLE, HotpotQA-derived data, CMRC, DuReader) are themselves older public benchmarks, and the full LV-Eval test set and its answers have been public and downloadable since February 2024 with no held-out portion, so exact-instance memorization by a model trained afterward is possible even though the authors' mitigations target a different, subtler leakage problem.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en",
    "zh"
   ],
   "license": "MIT for the code and packaging (GitHub LICENSE file and the Hugging Face license tag); the GitHub repository also ships a separate LICENSE_CC file (CC BY-SA 4.0) without stating which of the two covers the released data itself versus the evaluation code.",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 8645,
   "size_note": "11 datasets share 1,331 unique question-answer pairs (596 single-hop, 735 multi-hop), each rendered as a separate graded context at all five length levels, for 8,645 total test instances (summed directly from the per-dataset context counts the authors publish). The two fact-recall datasets (factrecall-en, factrecall-zh) hold only one QA pair each, repeated across 200 differently arranged contexts per length level, as a dedicated \"needle in a haystack\" pressure test. The Hugging Face dataset card's own description text says \"12 finegrained tasks,\" but only 11 are named in the paper, in the README's task tables, and as ZIP files in the repository; that description text appears to be an error in the card rather than a 12th task this page could locate.\n",
   "splits": "test only, no train/validation split; each of the 11 datasets is pre-split into five length-level variants (e.g. hotpotwikiqa_mixup_16k ... _256k)",
   "url": "https://huggingface.co/datasets/Infinigence/LVEval"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 3, slice A",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "lveval (per-dataset configs under opencompass/configs/datasets/lveval/, e.g. lvevalhotpotwikiqa_mixup, lvevalfactrecall_en; no lm-evaluation-harness, inspect_evals or HELM implementation was found during this research)",
   "other": "The authors' own prediction.py / evaluation.py scripts in infinigence/LVEval are the reference implementation, with batch shell scripts for data-parallel or single-GPU evaluation and a separate script for API-based commercial models."
  },
  "id": "lveval",
  "last_updated": "2024-08",
  "leaderboard_url": "https://github.com/infinigence/LVEval",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "LV-Eval gives a model a long bilingual (English/Chinese) document, built by mixing real supporting passages with distracting ones, and asks a single-hop or multi-hop question over it. It spans 11 datasets -- six single-hop (including a fact-recall pressure test styled on needle-in-a-haystack) and five multi-hop, several adapted from sources such as LooGLE, HotpotQA-derived data, CMRC and DuReader -- each rendered at five length levels (16k, 32k, 64k, 128k, 256k words) using the same underlying question-answer pairs, so a model's degradation curve across lengths can be measured directly rather than compared across different questions. Most datasets additionally insert GPT-4-generated, human-revised \"confusing facts\" into the context and apply keyword-and-phrase replacement, so a model must reason from the given text rather than lean on memorized or common-sense knowledge, which the authors argue inflates scores on benchmarks built from unaltered public documents.\n",
  "metric": {
   "baseline_note": "LV-Eval does not establish a fixed random-guess or human baseline; it is designed instead for relative, controlled comparison of the same model across its five length levels and against other evaluated models. The paper's own reference points are other LLMs: at release, the strongest 2023-era open model tested, ChatGLM3-6B-32k, scored 30.70% at the easiest (16k) level, falling to 7.17% at 256k.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "keyword-recall-based F1 (most datasets), F1 with word blacklist (cmrc-mixup), or ROUGE-L with word blacklist (dureader-mixup); averaged per length level and across datasets",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "LV-Eval",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2402.05136",
   "title": "LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K",
   "url": "https://arxiv.org/abs/2402.05136",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Tao Yuan",
    "Xuefei Ning",
    "Dong Zhou",
    "Zhijie Yang",
    "Shiyao Li",
    "Minghui Zhuang",
    "Zheyue Tan",
    "Zhuyu Yao",
    "Dahua Lin",
    "Boxun Li",
    "Guohao Dai",
    "Shengen Yan",
    "Yu Wang"
   ],
   "org": "Tsinghua University; Infinigence-AI; Shanghai Jiao Tong University; The Chinese University of Hong Kong; Shanghai Artificial Intelligence Laboratory",
   "url": "https://github.com/infinigence/LVEval"
  },
  "released": "2024-02",
  "repo_url": "https://github.com/infinigence/LVEval",
  "saturation": {
   "as_of": "",
   "note": "Not saturated. The originally published (2024) reference table shows every 7B-scale model well below 50% even at the easiest 16k level, falling into single digits by 256k. The paper's most recent text revision (v3, Oct 2025, per arXiv's own revision history) reports a wider 15-model sweep including Moonshot-v1-128k, Qwen2.5-72B-Instruct and Llama-3.1-70B-Instruct, stating that \"both Qwen2.5-72B and Moonshot-v1 obtain average scores exceeding 40 at 16k and 32k lengths\" -- still well short of a ceiling, and this page could not locate an exact top-score figure for that expanded run, only this qualitative statement, so top_score is left unset rather than estimated. The GitHub repository itself has not been pushed to since August 2024, so this newer 15-model comparison exists only in the revised paper text, not as a rerunnable reference table.\n",
   "status": "open",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "LV-Eval: A Balanced Long-Context Benchmark with 5 Length Levels Up to 256K (Yuan et al., arXiv:2402.05136)",
    "url": "https://arxiv.org/abs/2402.05136"
   },
   {
    "accessed": "2026-09-08",
    "title": "LV-Eval, full text (ar5iv, reflecting the Oct 2025 v3 revision)",
    "url": "https://ar5iv.labs.arxiv.org/html/2402.05136"
   },
   {
    "accessed": "2026-09-08",
    "title": "infinigence/LVEval GitHub repository (README, LICENSE, LICENSE_CC, evaluation scripts)",
    "url": "https://github.com/infinigence/LVEval"
   },
   {
    "accessed": "2026-09-08",
    "title": "Infinigence/LVEval dataset card and API metadata, Hugging Face",
    "url": "https://huggingface.co/datasets/Infinigence/LVEval"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass lveval dataset configs (11 per-dataset subdirectories)",
    "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/lveval"
   }
  ],
  "status": "active",
  "subcategory": "bilingual long-context QA with confusion injection and keyword-recall scoring, five length levels to 256k words",
  "summary": "A bilingual long-context QA benchmark with 11 datasets at five length levels from 16k to 256k words, built to fight knowledge leakage with confusing-fact insertion and keyword-recall metrics.",
  "tags": [
   "long-context",
   "bilingual",
   "question-answering",
   "multi-hop",
   "needle-in-a-haystack",
   "knowledge-leakage",
   "keyword-recall"
  ],
  "task_format": "A long document (16k-256k words) mixing supporting and distracting passages, sometimes with inserted confusing facts, plus a single-hop or multi-hop question; a short free-form answer out, evaluated with greedy decoding. Ten of the 11 datasets are scored with a two-stage keyword-recall metric built from manually annotated answer keywords and a blacklist of non-informative words; the remaining two (cmrc-mixup, dureader-mixup) use a plain F1 or ROUGE-L variant with the same word blacklist.\n"
 }
}