Natural Questions (HELM)

HELM's Natural Questions wrap: short answers to real Google searches, in closed-book, long-answer-context, or full-Wikipedia-page modes.

Also known as: NaturalQA, NaturalQuestions, natural_qa_closedbook, natural_qa_openbook_longans, natural_qa_openbook_wiki

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	knowledge
Subcategory	short-answer QA on real Google queries, with optional Wikipedia context
Page status	active
Metric	token F1 against the set of short answers (HELM f1_score)
Direction	higher_is_better
Unit	%
Dataset size	7830
Dataset licence	Apache-2.0 (repository LICENSE); Wikipedia page text remains under Wikipedia's licence
Publisher	Google Research; HELM scenario by Stanford CRFM

What it measures

This id is HELM's natural_qa scenario (Kwiatkowski et al., TACL 2019), not the original NQ long-answer span-selection competition and not lm-eval nq_open. Each example is a real anonymized Google query plus a Wikipedia page. HELM keeps only items with at least one short answer, drops yes/no, and asks the model to generate a short string. Three context modes exist: closedbook (question only), openbook_longans (annotator long answer as passage), openbook_wiki (title plus full page). English text.

Task format

Short-answer generation. HELM Lite run spec natural_qa:mode={closedbook, openbook_longans, openbook_wiki}, output noun Answer, max_tokens=300. Closed-book uses input noun Question. Main split in the schema is valid; metric f1_score.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub