lm-evaluation-harness's squadv2 task runs the SQuAD 2.0 validation set, scoring exact match and F1 separately for answerable and unanswerable questions via the official squad_v2 metric.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | extractive reading comprehension with unanswerable questions, as implemented by lm-evaluation-harness's squadv2 task |
| Page status | saturated |
| Metric | Hugging Face evaluate 'squad_v2' metric: exact match and F1 overall, plus HasAns_exact/f1 (answerable-only), NoAns_exact/f1 (unanswerable-only) and best_exact/best_f1 (threshold-optimised) submetrics |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 11873 |
| Dataset licence | CC BY-SA 4.0 |
| Publisher | Stanford University (original dataset); EleutherAI maintains this harness implementation |
This is the lm-evaluation-harness implementation of SQuAD 2.0 (task name `squadv2`): given a Wikipedia passage and a question, the model must generate the answer text found in the passage, or the harness estimates a "no answer" probability from loglikelihoods when the question is one of the adversarially written unanswerable questions that make up SQuAD 2.0's added difficulty over SQuAD 1.1. See [SQuAD](squad.md) for the full history and dataset detail shared by both SQuAD versions; this page documents the specific harness variant that model cards in this repository score under the id `squadv2`.
lm-evaluation-harness loads the `lighteval/squad_v2` mirror's validation split and formats each item as "Title: {title}\n\nBackground: {context}\n\nQuestion: {question}\n\nAnswer:", generating a free-text answer; for unanswerable items it estimates a no_answer_probability from the model's loglikelihoods rather than requiring a literal refusal string.
No model card in ModelSpec reports this benchmark yet.