Student–teacher dialogs over a hidden Wikipedia section; HELM scores free-text F1 at a random turn, while the official board uses span F1 on a hidden test set.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | information-seeking dialog QA over a Wikipedia section |
| Page status | active |
| Metric | F1 (word overlap; HELM f1_score on valid; official also HEQ-Q / HEQ-D) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 98407 |
| Dataset licence | CC-BY-SA-4.0 (quac.ai download page). Hugging Face allenai/quac tags MIT, which disagrees with the official page. |
| Publisher | Allen Institute for AI, University of Washington, Stanford University, UMass Amherst |
QuAC tests conversational reading of a Wikipedia section about a person. A student crowd worker sees only the article title and the lead paragraph and asks a sequence of questions. A teacher, who can see the section, answers with a short span (at most 30 tokens) or CANNOTANSWER, plus dialog acts for follow-up and yes/no. Later questions depend on earlier turns, so the set exercises coreference and open-ended information seeking rather than SQuAD-style paraphrase of a visible paragraph.
HELM's QuACScenario builds a prompt with title, background, section heading, passage, and a prefix of the dialog, then asks for the next answer. Official evaluation is span F1 / HEQ on a hidden test set with five references on dev and test.
No model card in ModelSpec reports this benchmark yet.