Closed-book short-answer QA on real Google queries from Natural Questions, usually scored by exact match on the public 3,610-item original NQ-Open dev split.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | closed-book short-answer QA on real Google queries derived from Natural Questions |
| Page status | active |
| Metric | exact match (lm-eval exact_match; official NQ-Open also uses EM against answer aliases) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 3610 |
| Dataset licence | CC BY-SA 3.0 (nq_open README and Hugging Face card); natural-questions GitHub LICENSE file is Apache-2.0 for the repository |
| Publisher | Google Research |
This id is EleutherAI lm-eval task nq_open and the NQ-Open task of Lee, Chang and Toutanova (ACL 2019). It is not HELM natural_qa and not the 2019 long-answer span-selection competition. Each item is an English question from aggregated Google searches. The model must emit a short answer string with no Wikipedia page in the prompt. Lee et al. keep Natural Questions items that have a short answer of at most five tokens and drop the evidence document. All questions are answerable from English Wikipedia in principle.
lm-eval generate_until. Prompt "Q: {question}?\nA:" after a few-shot header "Answer these questions:\n\n". Stops at newline, period or comma. Greedy decoding (temperature 0). English text.
No model card in ModelSpec reports this benchmark yet.