GLUE's largest task: judge whether two Quora questions ask the same thing, scored by the mean of accuracy and F1 because the classes are unbalanced.
unassessed
| Category | composite |
|---|---|
| Subcategory | duplicate question detection (sentence pair, English community Q&A) |
| Page status | superseded |
| Metric | mean of accuracy and F1 |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 40430 |
| Dataset licence | Dataset card licence: other; see the glue family page for the composite-licence explanation. |
| Publisher | New York University |
Two English questions posted to the community question-and-answer site Quora, with a label for whether they are duplicates -- semantically asking the same thing -- or not. QQP is GLUE's largest task by far: hundreds of thousands of question pairs, versus low thousands for most of the rest of the suite. The classes are imbalanced (63% negative, per the GLUE paper), which is why, as with MRPC, GLUE scores QQP by the mean of accuracy and F1 rather than accuracy alone.
Binary sentence-pair classification (duplicate / not duplicate), commonly cast by harnesses as a two-way multiple-choice or yes/no generation task. Scored by the mean of accuracy and F1.
No model card in ModelSpec reports this benchmark yet.