XNLI (Cross-lingual Natural Language Inference)

Cross-lingual three-way NLI in 15 languages: given a premise and hypothesis, choose entailment, contradiction, or neutral.

Also known as: Cross-lingual NLI, facebook/xnli

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	reasoning
Subcategory	15-language three-way textual entailment
Page status	active
Metric	accuracy
Direction	higher_is_better
Unit	%
Dataset size	112500
Dataset licence	CC BY-NC 4.0
Publisher	Facebook AI, with New York University

What it measures

XNLI tests whether a model can decide, in each of 15 languages, if a premise sentence entails a hypothesis, contradicts it, or neither (neutral). The English pairs come from MultiNLI's development and test genres. Professional translators produced the other 14 languages so the same labels apply across the parallel set. The original use was cross-lingual transfer: train NLI in English, test in another language without target-language NLI labels.

Task format

Three-way classification (entailment / neutral / contradiction). lm-evaluation-harness scores multiple-choice log-likelihood over a cloze: premise + ", {right}? {Yes|Also|No}, " + hypothesis, with those cue words translated per language in utils.py. The group `xnli` averages accuracy across 15 language tasks, weighted by size.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub