{
 "body": "\n## What it measures\n\nTriviaQA was built around 95,956 question-answer pairs authored by trivia enthusiasts, each paired\nwith evidence documents (six per question on average, drawn from Wikipedia and general web search\nresults) gathered automatically to provide distant supervision. The paper's own task is reading\ncomprehension: given a question and one or more evidence documents, extract the answer span from the\ntext -- explicitly positioned by the authors as harder than SQuAD, with more compositional\nquestions, more lexical variation between question and evidence, and more need for cross-sentence\nreasoning. Since roughly 2019, most LLM evaluation instead uses the same question-answer pairs with\nthe evidence stripped out entirely: an open-domain, closed-book setting where the model must produce\nthe answer purely from what it learned during training, with no document to read from. These test\ndifferent things -- extraction versus recall -- and a bare \"TriviaQA\" score does not say which one\nwas measured.\n\n## How it is scored\n\nIn the paper's own reading-comprehension setting, a prediction is scored with SQuAD-style Exact\nMatch (EM) and F1 against a list of accepted answer aliases (Wikipedia-entity answers get a full\nlist of alternate titles; numerical and free-form answers use a single reference string). In the\nopen-domain setting that lm-evaluation-harness and most current LLM papers actually run, no document\nis shown at all: the model is prompted with only the question and graded by Exact Match against the\nsame alias list, evaluated on the validation split since the real test-set answers were never\nreleased. Confirmed directly from lm-evaluation-harness's task config: it reads the `rc.nocontext`\nversion of the dataset -- the reading-comprehension question set with its evidence removed -- not\nthe separate, larger \"unfiltered\" collection the authors built specifically for open-domain research\n(see Dataset and licence).\n\n## Dataset and licence\n\nThe headline dataset statistic is 95,956 question-answer pairs, 662,659 evidence documents and\n40,478 unique answers. The released reading-comprehension package instead counts by domain-specific\nquestion instances: 61,888/7,993/7,701 (train/validation/test) for the Wikipedia domain and\n76,496/9,951/9,509 for the Web domain, both confirmed against the current Hugging Face release and\nmatching the paper's own published table exactly. A separate, larger \"unfiltered\" release (110,495\nQA pairs per the paper; currently hosted as 87,622/11,313/10,832 train/validation/test) exists\nspecifically for open-domain and IR-style research, where not every paired document is guaranteed to\ncontain the answer. A much smaller, human-verified subset (297-733 questions per split) marks cases\nwhere an annotator confirmed the evidence genuinely supports the answer. The GitHub repository states\nan Apache 2.0 licence covers both code and data; the official site separately notes the University of\nWashington does not own the copyright of the underlying questions and documents, sourced from\nthird-party trivia sites, Wikipedia and web search results.\n\n## Who publishes it\n\nTriviaQA was introduced by Mandar Joshi, Eunsol Choi, Daniel S. Weld and Luke Zettlemoyer of the\nUniversity of Washington's Paul G. Allen School (Zettlemoyer also affiliated with the Allen\nInstitute for Artificial Intelligence), presented at ACL 2017. The authors maintain the reference\ndata, evaluation code and project site; the reading-comprehension test set was historically scored\nthrough a CodaLab leaderboard, per the official site, though this page could not confirm whether that\nleaderboard remains active today.\n\n## Lineage\n\nTriviaQA has no formal predecessor or successor as a benchmark id; the paper positions it as a\nharder, more naturally-sourced alternative to SQuAD and similar single-paragraph reading-comprehension\ndatasets available at the time. Its own \"unfiltered\" release, built specifically to support\nopen-domain and IR-style question answering, is best understood as an internal variant of the same\nproject rather than a separate benchmark, and predates by several years the closed-book usage that\nlater became the dominant way LLM papers report \"TriviaQA\" scores.\n\n## Saturation and contamination\n\nThe paper's own original result -- BiDAF at 40% EM in the reading-comprehension setting, against\nroughly 80% for a human given the same evidence -- is a 2017 baseline with no bearing on current\nmodels. This benchmark's age and heavy reliance on Wikipedia-derived facts make it a plausible\ncandidate for saturation at the frontier in its open-domain form, but this page could not confirm a\nspecific, currently dated top score from a source opened during this research, so saturation status\nis left unknown rather than guessed. Contamination risk is graded high: the dataset has been fully\npublic with answers since 2017, is one of the most widely cited QA benchmarks in NLP, and draws heavily on\nWikipedia content that overlaps extensively with the training data of essentially every\nbroad-coverage language model; lm-evaluation-harness's own task config includes a built-in\ndecontamination check, itself a sign the community treats this as a live concern.\n\n## How to run it\n\nThe original reference evaluation (`evaluation.triviaqa_evaluation` in mandarjoshi90/triviaqa on\nGitHub) implements the paper's own SQuAD-style EM/F1 scoring against a given evidence document.\nlm-evaluation-harness's `triviaqa` task instead reads the `rc.nocontext` configuration -- no evidence\nshown -- generates freely, and scores with case- and punctuation-insensitive Exact Match against the\nquestion's answer aliases, evaluated on the validation split. OpenCompass ships its own `triviaqa`\nconfiguration. Because harnesses differ in which configuration and split they use, always confirm\nwhich protocol produced a reported \"TriviaQA\" number before comparing it to another.\n\n## Reading the numbers\n\nA high TriviaQA score means different things depending on the setting: in the open-domain,\nclosed-book form most LLM papers report, it shows the model can recall a specific trivia fact from\nits own training without any supporting text, a reasonably direct proxy for breadth of memorized\nworld knowledge; in the original reading-comprehension form, it instead shows the model can locate\nand extract an answer already present in a given document, closer to a retrieval-and-extraction\nskill. Because the dataset is old, fully public, and heavily Wikipedia-derived, a very high\nclosed-book score is at least as likely to reflect memorization of this specific benchmark as broad\nfactual competence, and is worth corroborating against a newer or decontaminated knowledge benchmark\nbefore treating it as strong evidence on its own.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "triviaqa",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "knowledge",
  "contamination": {
   "note": "The dataset has been fully public, with train and validation answers included, since 2017, and has been one of the most widely cited QA benchmarks in NLP for years; its evidence documents draw heavily on Wikipedia, which overlaps enormously with the pretraining corpora of essentially every broad-coverage language model. lm-evaluation-harness's own task config for this benchmark enables a built-in decontamination check (`should_decontaminate: true`, matching against the question text), which is itself evidence that the community treats contamination as a live concern for this specific task.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Apache 2.0 for the code and the released data package, per the GitHub repository's own statement that this licence \"applies to both the code and the data.\" The project's official site separately notes that \"the University of Washington does not own the copyright of the questions and documents included in TriviaQA,\" since the underlying trivia questions and evidence text were sourced from third-party trivia sites, Wikipedia and web search results -- both statements are reported here rather than reconciled.\n",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": 95956,
   "size_note": "95,956 question-answer pairs is the paper's own headline count (Table 2), alongside 662,659 evidence documents and 40,478 unique answers. Two further wrinkles affect any \"size\" claim: (1) the released reading-comprehension package splits by domain rather than by unique question -- the Wikipedia domain has 61,888/7,993/7,701 train/validation/test question instances and the Web domain has 76,496/9,951/9,509 (some questions recur across domains, and Wikipedia-domain questions can pair with multiple evidence documents), both confirmed via the Hugging Face datasets-server API and matching the paper's own Table 6 exactly; (2) the authors separately released a larger \"unfiltered\" version (110,495 QA pairs per the paper) explicitly to support open-domain and IR-style QA research, where not every paired document is guaranteed to contain the answer; the currently hosted `unfiltered.nocontext` config totals 87,622/11,313/10,832 train/validation/test, close to but not an exact match to the paper's original figure. A much smaller human-verified subset also exists: 297/584 (Wikipedia dev/test) and 322/733 (Web dev/test) questions where a human annotator confirmed the evidence document actually supports the answer.\n",
   "splits": "Train / validation / test per domain and per configuration (see size_note). The public test split's answers are placeholders (`<unk>`, empty alias lists), confirmed directly by inspecting a test-split row via the Hugging Face datasets-server API -- the real test answers were never publicly released, historically scored instead through a CodaLab leaderboard (per the official project site). In practice, essentially every current harness and paper evaluates on the validation split as a proxy test set.\n",
   "url": "https://huggingface.co/datasets/mandarjoshi/trivia_qa"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 2, slice B",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "triviaqa (dataset_name: rc.nocontext -- open-domain, no evidence shown; exact_match against answer aliases; evaluated on the validation split; confirmed directly in the task config, including built-in decontamination support)",
   "opencompass": "triviaqa",
   "other": "The original reference evaluation (mandarjoshi90/triviaqa on GitHub, `evaluation.triviaqa_evaluation`) implements the paper's own SQuAD-style EM/F1 scoring against a given evidence document -- the reading-comprehension setting -- and was historically paired with a CodaLab leaderboard for the hidden test set, per the official project site. This is a materially different protocol from the no-context, closed-book setting lm-evaluation-harness and most current LLM papers report; the two are not comparable without knowing which one produced a given number."
  },
  "id": "triviaqa",
  "last_updated": "2024-01",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "TriviaQA was built around 95,956 question-answer pairs authored by trivia enthusiasts, each paired with evidence documents (six per question on average, drawn from Wikipedia and general web search results) gathered automatically to provide distant supervision. The paper's own task is reading comprehension: given a question and one or more evidence documents, extract the answer span from the text. Since roughly 2019, most LLM evaluation instead uses the same question-answer pairs with the evidence stripped out entirely -- an open-domain, closed-book setting where the model must produce the answer from its own parametric knowledge, with no document to read from. These are different tasks measuring different things, and a \"TriviaQA\" score can mean either one.\n",
  "metric": {
   "baseline_note": "The paper's abstract reports two baseline systems well below human performance in the reading-comprehension setting: a feature-based classifier at 23% and a SQuAD-strength neural network (BiDAF) at 40%, against roughly 80% for a human given the same evidence document -- the 80% figure comes from the same annotation exercise used to build the \"verified\" subset (below), not a separately run human study. No human baseline was established for the open-domain, no-context setting, since that use case was not the paper's own focus; \"random_baseline\" is left unset because this is free-text generation, not multiple choice.\n",
   "direction": "higher_is_better",
   "human_baseline": 80,
   "max_score": 100,
   "name": "Exact Match (EM) against answer aliases; F1 also used in the reading-comprehension setting",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "TriviaQA",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "1705.03551",
   "title": "TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension",
   "url": "https://arxiv.org/abs/1705.03551",
   "year": 2017
  },
  "publisher": {
   "authors": [
    "Mandar Joshi",
    "Eunsol Choi",
    "Daniel S. Weld",
    "Luke Zettlemoyer"
   ],
   "org": "University of Washington (Paul G. Allen School of Computer Science & Engineering); one author also affiliated with the Allen Institute for Artificial Intelligence",
   "url": "http://nlp.cs.washington.edu/triviaqa/"
  },
  "released": "2017-05",
  "repo_url": "https://github.com/mandarjoshi90/triviaqa",
  "saturation": {
   "as_of": "",
   "note": "The paper's own original result -- BiDAF at 40% EM in the reading-comprehension setting, well below the roughly 80% human figure -- is a 2017 baseline and not informative about current models. This benchmark is widely treated in the LLM literature as a standard open-domain, closed-book knowledge-recall check, and its age and Wikipedia-heavy sourcing make it a plausible candidate for saturation at the frontier, but this page could not confirm a specific current top score from a live, dated source: Epoch AI's benchmarking hub lists TriviaQA as a tracked benchmark but did not yield a specific figure through the pages checked, and a Papers with Code leaderboard lookup did not return a usable result. Status is left \"unknown\" rather than guessed.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension",
    "url": "https://arxiv.org/abs/1705.03551"
   },
   {
    "accessed": "2026-09-08",
    "title": "mandarjoshi90/triviaqa repository (README, licence statement)",
    "url": "https://github.com/mandarjoshi90/triviaqa"
   },
   {
    "accessed": "2026-09-08",
    "title": "TriviaQA official project site (RC vs. open-domain/unfiltered downloads, copyright note, CodaLab leaderboard history)",
    "url": "http://nlp.cs.washington.edu/triviaqa/"
   },
   {
    "accessed": "2026-09-08",
    "title": "mandarjoshi/trivia_qa dataset (8 configs: rc, rc.nocontext, rc.web[.nocontext], rc.wikipedia[.nocontext], unfiltered[.nocontext])",
    "url": "https://huggingface.co/datasets/mandarjoshi/trivia_qa"
   },
   {
    "accessed": "2026-09-08",
    "title": "mandarjoshi/trivia_qa configs and splits, Hugging Face datasets-server",
    "url": "https://datasets-server.huggingface.co/splits?dataset=mandarjoshi/trivia_qa"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness: triviaqa task config (rc.nocontext, exact_match, decontamination)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/triviaqa/default.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass dataset configs (includes triviaqa)",
    "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/triviaqa"
   },
   {
    "accessed": "2026-09-08",
    "title": "Epoch AI benchmarking hub (lists TriviaQA as a tracked benchmark)",
    "url": "https://epoch.ai/benchmarks"
   }
  ],
  "status": "active",
  "subcategory": "open-domain trivia recall; originally released as extractive reading comprehension over given evidence",
  "summary": "Trivia questions with answer-alias lists, run two very different ways -- extractive reading comprehension over given evidence, or open-domain closed-book recall -- with very different scores.",
  "tags": [
   "open-domain-qa",
   "trivia",
   "reading-comprehension",
   "exact-match",
   "knowledge-recall"
  ],
  "task_format": "Reading-comprehension setting (the paper's own task): given a question plus a Wikipedia or web evidence document, extract an answer span, scored with SQuAD-style Exact Match (EM) and F1 against a list of accepted answer aliases. Open-domain setting (how lm-evaluation-harness and most current LLM papers run it): given only the question, generate an answer with no document shown, scored by Exact Match against the same alias list -- confirmed directly from lm-evaluation-harness's task config, which reads the `rc.nocontext` version of the dataset (reading-comprehension questions with context removed, not the separate \"unfiltered\" open-domain release described below).\n"
 }
}