{
 "body": "\n## What it measures\n\nLIT-RAGBench evaluates the generator portion of retrieval-augmented generation. The input includes retrieved context and a question, and the benchmark tests whether the model can integrate that evidence into an answer, reason over it, handle logical questions, read tabular information, and abstain when the context does not support an answer. The paper defines five categories: Integration, Reasoning, Logic, Table, and Abstention.\n\nThe source describes 114 questions constructed by humans in Japanese, together with an English version produced by machine translation and then curated by humans. This makes the suite useful for comparing the same RAG generator capabilities across the two languages, while leaving the exact per-language item accounting to the released benchmark materials.\n\n## How it is scored\n\nThe paper uses an LLM-as-a-Judge procedure and reports category-wise and overall accuracy. The opened abstract does not specify the judge model, prompt, normalization, or aggregation details needed for exact reproduction, so those details should accompany any reported number. No random or human baseline was established in the source reviewed here.\n\nThe paper reports that none of the evaluated models exceeded 90% overall. That observation describes the experiment in the paper; it is not a permanent leaderboard ceiling and should not be treated as a current saturation claim.\n\n## Dataset and licence\n\nThe benchmark consists of text questions paired with RAG evaluation contexts and answer requirements. The paper abstract establishes the 114-question Japanese collection and the curated English version, but it does not state a train/dev/test split or a dataset licence. Confirm the release files and terms before redistributing the data.\n\n## Who publishes it\n\nLIT-RAGBench is described in the 2026 paper by Koki Itai, Shunichi Hasegawa, Yuta Yamamoto, Gouki Minegishi, and Masaki Otsuki. The arXiv record shows an initial submission in March 2026 and a revised version in April 2026. The opened source does not provide a stable repository or leaderboard URL.\n\n## Lineage\n\nThe benchmark belongs to the RAG evaluation family and focuses on generation quality after retrieval. No predecessor, successor, or stable variant was established from the source opened for this page.\n\n## Saturation and contamination\n\nThe question set is public through the paper's release context, so memorization or exposure to the benchmark can affect results. The paper does not establish a rotating private holdout or a contamination study in the source reviewed here. Its reported sub-90% model results indicate remaining headroom in that experiment, while the current saturation status remains unknown.\n\n## How to run it\n\nUse the authors' released evaluation materials when available, supplying the prescribed retrieved context and recording the exact generator, retrieval context, judge model, judge prompt, and language version. The opened arXiv record does not identify an lm-evaluation-harness, HELM, Inspect, OpenCompass, or BIG-bench task name.\n\n## Reading the numbers\n\nHigher category and overall accuracy indicate more judged answers meeting the benchmark's requirements. Compare Japanese and English results only with the same context construction, model instructions, judge model, and aggregation. Because an LLM judge is part of the metric, scores also reflect judge behavior and prompt choices; they should not be read as a direct measure of retrieval quality alone.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "lit_ragbench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "The question set and paper are public. The opened source does not describe a private or rotating test set or a contamination study.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "ja",
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 114,
   "size_note": "The paper describes 114 human-constructed Japanese questions and an English version produced by machine translation and human curation; the source does not state a separate split count.",
   "splits": "No train/dev/test split is specified in the opened paper abstract.",
   "url": "https://arxiv.org/abs/2603.06198v2"
  },
  "freshness": {
   "researched": "2026-09-09",
   "researched_by": "GPT-5.6 Luna, luna-stream-c-004 (Codex coordinated)",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "The paper describes an evaluation implementation with LLM-as-a-Judge scoring; a stable public harness identifier was not established from the opened abstract."
  },
  "id": "lit_ragbench",
  "last_updated": "2026-04",
  "leaderboard_url": "",
  "lineage": {
   "family": "RAG evaluation",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "LIT-RAGBench tests whether a language model can use supplied retrieved context to generate an answer, including recognizing when it should abstain. Its five categories are Integration, Reasoning, Logic, Table, and Abstention, with Japanese questions and an English machine-translated and human-curated version.\n",
  "metric": {
   "baseline_note": "The paper describes category-wise and overall accuracy judged by an LLM; it does not establish a single random or human baseline.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "LLM-as-a-Judge accuracy",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "LIT-RAGBench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2603.06198",
   "title": "LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation",
   "url": "https://arxiv.org/abs/2603.06198v2",
   "year": 2026
  },
  "publisher": {
   "authors": [
    "Koki Itai",
    "Shunichi Hasegawa",
    "Yuta Yamamoto",
    "Gouki Minegishi",
    "Masaki Otsuki"
   ],
   "org": "LIT-RAGBench authors",
   "url": "https://arxiv.org/abs/2603.06198v2"
  },
  "released": "2026-04",
  "repo_url": "",
  "saturation": {
   "as_of": "",
   "note": "The paper reports that no evaluated model exceeded 90% overall, but this is a study result rather than evidence of a current saturation ceiling.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-09",
    "title": "LIT-RAGBench paper and abstract",
    "url": "https://arxiv.org/abs/2603.06198v2"
   }
  ],
  "status": "active",
  "subcategory": "retrieval-augmented generation answer evaluation",
  "summary": "LIT-RAGBench evaluates retrieval-augmented generation generators on 114 human-constructed Japanese questions and a curated English version across integration, reasoning, logic, table and abstention capabilities.",
  "tags": [
   "rag",
   "retrieval-augmented-generation",
   "question-answering",
   "abstention",
   "japanese",
   "english"
  ],
  "task_format": "Context-grounded question answering with category-specific answer generation and abstention cases."
 }
}