Adversarial restatement of five GLUE tasks using word-level, sentence-level and human-written attacks; OpenCompass scores a public annotated development pack, not the original hidden test set.
unassessed
| Category | composite |
|---|---|
| Subcategory | adversarial robustness suite over five GLUE tasks (SST-2, QQP, QNLI, RTE, MNLI) |
| Page status | active |
| Metric | per-task GLUE metrics averaged as an AdvGLUE score; OpenCompass additionally reports acc_drop = 1 - (adversarial accuracy / original accuracy) |
| Direction | higher_is_better |
| Dataset size | 4978 |
| Dataset licence | CC-BY-SA-4.0 |
| Publisher | University of Illinois Urbana-Champaign and Microsoft |
AdvGLUE asks whether a model that looks strong on GLUE still labels the same sentence-pair and sentiment tasks after adversarial edits. It covers five GLUE tasks: SST-2 sentiment, QQP paraphrase, and the NLI tasks MNLI, QNLI and RTE. CoLA and WNLI from GLUE are omitted. Attacks include 14 methods grouped as word-level (embedding similarity, typos, context-aware, knowledge-guided, compositions), sentence-level (syntactic and distraction), and human-crafted sets (CheckList, StressTest, ANLI, AdvSQuAD). Crowd workers filtered examples so that the label should still hold.
Same classification or NLI format as the parent GLUE task, on adversarially perturbed text. OpenCompass instead prompts for an option letter and reports accuracy drop against concatenated original items.
No model card in ModelSpec reports this benchmark yet.