{
 "body": "\n## What it measures\n\nOpenCompass passes an evidence document and a question to a model and requests an answer, testing reading comprehension rather than closed-book recall. The prompt template is confirmed directly from the configuration: `\"{evidence}\\nAnswer these questions:\\nQ: {question}?A:\"`, with zero demonstration examples (`ZeroRetriever`).\n\n## How it is scored\n\nThe configuration uses `TriviaQAEvaluator` with generated answers, run zero-shot with no in-context examples. Exact normalization details (how the evaluator matches generated text against the answer-alias list) are not established from the configuration file alone.\n\n## Dataset and licence\n\nThe configuration uses the dev split for both its train and test fields and a local `triviaqa-rc` path. It does not state item count or licence.\n\n## Who publishes it\n\nOpenCompass maintains the runnable integration. TriviaQA was introduced by Joshi and colleagues in the cited paper.\n\n## Lineage\n\nThis is an OpenCompass configuration built on [TriviaQA](triviaqa.md), the Joshi et al. dataset cited above. TriviaQA is run two very different ways in the literature: the paper's own reading-comprehension setting (question plus evidence document, extract the answer), which this configuration uses, and a more common open-domain closed-book setting (question only, no evidence) used by lm-evaluation-harness and most current LLM papers. Scores from this page should not be merged with open-domain TriviaQA scores; see the `triviaqa` page for that distinction.\n\n## Saturation and contamination\n\nSaturation is unknown. Public evidence and answers create plausible contamination risk.\n\n## How to run it\n\nRun OpenCompass with dataset abbreviation `triviaqarc`; the configuration uses `ZeroRetriever` (no few-shot examples), a maximum output length of 50 tokens, and the `dev` split for both its train and test fields, loading from a local `./data/triviaqa-rc/` path rather than a Hugging Face dataset id. Record the OpenCompass config revision (`triviaqarc_gen_db6413`), since OpenCompass versions its dataset configs by hash suffix and other revisions may use different prompts or evaluators.\n\n## Reading the numbers\n\nA strong score indicates answer generation from supplied evidence under this configuration. It does not establish broad knowledge without evidence. Compare evaluator normalization and evidence split, and do not confuse this reading-comprehension score with the open-domain closed-book TriviaQA score most current LLM papers report.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "triviaqarc",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "knowledge",
  "contamination": {
   "note": "TriviaQA questions and evidence are public; training exposure is plausible.",
   "risk": "unknown"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "The OpenCompass configuration does not state a total item count.",
   "splits": "dev",
   "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/triviaqarc"
  },
  "freshness": {
   "luna-new-002": null,
   "luna-new-002 (Codex coordinated)": null,
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "triviaqarc",
   "other": ""
  },
  "id": "triviaqarc",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "triviaqa",
   "successors": [],
   "variants": []
  },
  "measures": "The task presents a question with evidence and asks the model to produce the answer.",
  "metric": {
   "baseline_note": "",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "exact match",
   "random_baseline": null,
   "unit": "percent"
  },
  "name": "TriviaQA RC",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "1606.05250",
   "title": "TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension",
   "url": "https://arxiv.org/abs/1606.05250",
   "year": 2016
  },
  "publisher": {
   "authors": [],
   "org": "OpenCompass",
   "url": "https://github.com/open-compass/opencompass"
  },
  "released": "",
  "repo_url": "https://github.com/open-compass/opencompass",
  "saturation": {
   "as_of": "",
   "note": "OpenCompass uses TriviaQAEvaluator; no current standalone leaderboard was established.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass TriviaQArc configuration",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/triviaqarc/triviaqarc_gen_db6413.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "TriviaQA paper",
    "url": "https://arxiv.org/abs/1606.05250"
   }
  ],
  "status": "active",
  "subcategory": "open-domain question answering",
  "summary": "TriviaQA reading comprehension evaluated through the OpenCompass TriviaQArc configuration.",
  "tags": [
   "question-answering",
   "reading-comprehension"
  ],
  "task_format": "Evidence passage, question, and generated answer."
 }
}