SuperGLUE's three-class entailment recast of CommitmentBank: 250/56/250 English pairs, scored with accuracy and macro-F1.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | three-class textual entailment from speaker commitment (SuperGLUE) |
| Page status | saturated |
| Metric | accuracy and macro-F1 (unweighted mean of per-class F1); SuperGLUE reports both |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 556 |
| Dataset licence | other |
| Publisher | New York University (SuperGLUE); original CommitmentBank from Ohio State / Carnegie Mellon / Ohio State authors |
SuperGLUE CB asks whether a premise commits its author to a hypothesis extracted from an embedded clause. The original CommitmentBank labels how committed a speaker is to that clause on a Likert scale. SuperGLUE recasts the items as three-class textual entailment: entailment, contradiction, or neutral. Premises come from the Wall Street Journal, British National Corpus fiction, and Switchboard. SuperGLUE keeps a subset with inter-annotator agreement above 80% and uses its own split. The task is English text, single-turn classification, not open-ended generation.
Three-way classification: premise plus hypothesis to entailment, contradiction, or neutral; English.
No model card in ModelSpec reports this benchmark yet.