OpenCompass's squad20 task runs the SQuAD 2.0 dev set, prompting a model to extract an answer span or say "impossible to answer" for adversarial unanswerable questions.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | extractive reading comprehension with unanswerable questions, as implemented by OpenCompass's squad20 config |
| Page status | saturated |
| Metric | SQuAD20Evaluator (Exact Match / F1 against reference answers, with a literal 'impossible to answer' string scored against unanswerable questions) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 11873 |
| Dataset licence | CC BY-SA 4.0 |
| Publisher | Stanford University (original dataset); OpenCompass (open-compass) maintains this harness implementation |
This is the OpenCompass harness implementation of SQuAD 2.0 (id `squad20`, config abbreviation `squad2.0`): given a Wikipedia passage and a question, the model must either return the answer text found in the passage or state that the question is impossible to answer, since over 50,000 of SQuAD 2.0's questions are adversarially written to closely resemble answerable ones while having no supported answer in the passage. See [SQuAD](squad.md) for the full history and dataset detail shared by both SQuAD 1.1 and 2.0; this page documents the specific harness variant that model cards in this repository score under the id `squad20`.
OpenCompass reads `dev-v2.0.json` locally and prompts: "According to the above passage, answer the following question. If it is impossible to answer according to the passage, answer 'impossible to answer'", followed by the question, with generation capped at 50 tokens.
No model card in ModelSpec reports this benchmark yet.