A 5-way multiple-choice commonsense test built from ConceptNet relations, designed so questions need world knowledge beyond the immediate text.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | commonsense multiple-choice question answering |
| Page status | saturated |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 12102 |
| Dataset licence | MIT, per the Hugging Face dataset card; the reference GitHub repository (jonathanherzig/commonsenseqa) carries no separate LICENSE file |
| Publisher | Tel Aviv University; Allen Institute for Artificial Intelligence |
CommonsenseQA tests whether a model can answer questions that require general world knowledge rather than information available in a supplied passage. Questions were constructed by extracting sets of target concepts from ConceptNet that share the same semantic relation to one source concept, then having crowd-workers write a question that mentions the source concept and discriminates between the target concepts as candidate answers -- a process meant to force questions with complex semantics rather than simple keyword association.
Five-way multiple-choice question answering: one correct answer plus four distractor concepts drawn from the same ConceptNet neighborhood, evaluated zero-shot or few-shot with no supporting passage. The dataset ships a main "Random split" (train/validation/test) used for most reported results, plus a secondary "Question token split" described in the paper for testing generalization to unseen question wording.
No model card in ModelSpec reports this benchmark yet.