A three-round adversarial natural language inference benchmark where annotators iteratively wrote examples to fool the strongest model trained on all prior rounds.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | adversarial natural language inference |
| Page status | unknown |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 169265 |
| Dataset licence | CC BY-NC 4.0 |
| Publisher | Facebook AI Research (FAIR), with the University of North Carolina at Chapel Hill |
ANLI tests natural language inference: given a short context passage and a one-sentence hypothesis, a model must decide whether the hypothesis is entailed by the context, contradicted by it, or neither (neutral). What separates ANLI from earlier inference datasets like SNLI and MNLI is not the task format but how its examples were built. Human annotators were shown a context and a target label, then wrote a hypothesis specifically intended to fool a state-of-the-art NLI model already trained on all prior data; an item only entered the dataset once it beat that model and passed a separate human-verification pass. Three rounds (R1, R2, R3) repeated this loop with progressively stronger target models and, from round three onward, a wider range of source genres beyond Wikipedia.
Three-way classification: given a context and a hypothesis, label the pair as entailment, contradiction, or neutral.
No model card in ModelSpec reports this benchmark yet.