Based and lm-eval rewrite 2,984 SQuAD validation items as next-token completions, scored by case-insensitive contains.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | SQuAD passages rewritten so the gold span is the last tokens of a statement |
| Page status | unknown |
| Metric | contains |
| Direction | higher_is_better |
| Dataset size | 2984 |
| Publisher | Hazy Research (Stanford); EleutherAI lm-evaluation-harness |
squad_completion is a recall-style reading task, not span extraction and not SQuAD 2.0 abstention. A passage is followed by a statement that ends with the answer. The model must continue the statement so the generation contains that gold span. Hazy Research built the items for the BASED paper by asking GPT-4 to rewrite question-answer pairs as statements that end with the answer, then dropping rewrites that failed that check. English text. Zero-shot generation.
generate_until: prompt is doc["text"] with surrounding whitespace stripped; gold is doc["value"] stripped. Decoding stops at a newline, max_gen_toks 48. Metric is contains: case-insensitive regex search of the gold string inside the continuation. lm-eval VERSION 1.
No model card in ModelSpec reports this benchmark yet.