GLUE's single-sentence task: judge whether an English sentence is grammatically acceptable, scored by Matthews correlation because the acceptable/unacceptable classes are unbalanced.
unassessed
| Category | composite |
|---|---|
| Subcategory | linguistic acceptability judgement (single-sentence, English) |
| Page status | superseded |
| Metric | Matthews correlation coefficient (MCC) |
| Direction | higher_is_better |
| Unit | correlation, -1 to 1 |
| Dataset size | 1043 |
| Dataset licence | Dataset card licence: other; see the glue family page for the composite-licence explanation. |
| Publisher | New York University |
One English sentence, drawn from books and journal articles on linguistic theory, and one judgement: is it a grammatically acceptable sentence of English, or not. Unlike most of the rest of GLUE, this is a single-sentence task -- there is no second sentence to compare it against. About 70% of the training examples are labelled acceptable (6,023 of 8,551, per the Hugging Face datasets-server's own column statistics), unbalanced enough that raw accuracy would reward always guessing the majority class, which is why GLUE scores CoLA by Matthews correlation coefficient instead of accuracy.
Binary single-sentence classification (acceptable / unacceptable), commonly cast by harnesses as a two-way multiple-choice or yes/no generation task. Scored by Matthews correlation coefficient (MCC), a -1-to-1 correlation measure, not by accuracy.
No model card in ModelSpec reports this benchmark yet.