Trivia questions with answer-alias lists, run two very different ways -- extractive reading comprehension over given evidence, or open-domain closed-book recall -- with very different scores.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | open-domain trivia recall; originally released as extractive reading comprehension over given evidence |
| Page status | active |
| Metric | Exact Match (EM) against answer aliases; F1 also used in the reading-comprehension setting |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 95956 |
| Dataset licence | Apache 2.0 for the code and the released data package, per the GitHub repository's own statement that this licence "applies to both the code and the data." The project's official site separately notes that "the University of Washington does not own the copyright of the questions and documents included in TriviaQA," since the underlying trivia questions and evidence text were sourced from third-party trivia sites, Wikipedia and web search results -- both statements are reported here rather than reconciled. |
| Publisher | University of Washington (Paul G. Allen School of Computer Science & Engineering); one author also affiliated with the Allen Institute for Artificial Intelligence |
TriviaQA was built around 95,956 question-answer pairs authored by trivia enthusiasts, each paired with evidence documents (six per question on average, drawn from Wikipedia and general web search results) gathered automatically to provide distant supervision. The paper's own task is reading comprehension: given a question and one or more evidence documents, extract the answer span from the text. Since roughly 2019, most LLM evaluation instead uses the same question-answer pairs with the evidence stripped out entirely -- an open-domain, closed-book setting where the model must produce the answer from its own parametric knowledge, with no document to read from. These are different tasks measuring different things, and a "TriviaQA" score can mean either one.
Reading-comprehension setting (the paper's own task): given a question plus a Wikipedia or web evidence document, extract an answer span, scored with SQuAD-style Exact Match (EM) and F1 against a list of accepted answer aliases. Open-domain setting (how lm-evaluation-harness and most current LLM papers run it): given only the question, generate an answer with no document shown, scored by Exact Match against the same alias list -- confirmed directly from lm-evaluation-harness's task config, which reads the `rc.nocontext` version of the dataset (reading-comprehension questions with context removed, not the separate "unfiltered" open-domain release described below).
No model card in ModelSpec reports this benchmark yet.