SuperGLUE COPA (Choice of Plausible Alternatives)

SuperGLUE packaging of COPA: pick the more plausible cause or effect of a one-sentence English premise from two alternatives.

Also known as: COPA, Choice of Plausible Alternatives, SuperGLUE_COPA

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	reasoning
Subcategory	binary causal commonsense: cause or effect of a premise (SuperGLUE)
Page status	saturated
Metric	accuracy
Direction	higher_is_better
Unit	%
Dataset size	1000
Dataset licence	other
Publisher	USC Institute for Creative Technologies and Indiana University (original COPA); SuperGLUE from New York University and collaborators

What it measures

SuperGLUE COPA is a two-choice causal commonsense task. The model reads one English premise sentence and a question that is either cause or effect, then picks which of two alternatives is more plausible. Items are hand-authored, not mined from exams. The original COPA paper (Roemmele, Bejan, and Gordon, 2011) wrote 1,000 such questions and split them 500/500 into development and test. SuperGLUE uses 400 train, 100 validation, and 500 test, matching the original test set and splitting the original development set. The language is English; the format is forced choice, not free-text explanation.

Task format

Two-choice classification: premise plus cause/effect cue and two alternatives; English.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub