GLUE's paraphrase task: judge whether two English news sentences mean the same thing, scored by the mean of accuracy and F1 because the classes are unbalanced.
unassessed
| Category | composite |
|---|---|
| Subcategory | paraphrase detection (sentence pair, English news) |
| Page status | superseded |
| Metric | mean of accuracy and F1 |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 408 |
| Dataset licence | Dataset card licence: other; see the glue family page for the composite-licence explanation. |
| Publisher | New York University |
Two English sentences, automatically pulled from online news sources, with a human annotation for whether they are semantically equivalent -- a paraphrase judgement, not a similarity score. The task is binary sentence-pair classification: given both sentences, decide equivalent or not. The classes are imbalanced (68% positive, per the GLUE paper), which is why GLUE scores MRPC by the mean of accuracy and F1 rather than accuracy alone.
Binary sentence-pair classification (equivalent / not equivalent), commonly cast by harnesses as a two-way multiple-choice or yes/no generation task. Scored by the mean of accuracy and F1.
No model card in ModelSpec reports this benchmark yet.