Machine-translated XNLI for 11 Indic languages; lm-evaluation-harness currently scores only Gujarati three-way NLI on Divyanshu/indicxnli.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | 11-language Indic three-way NLI (lm-eval currently Gujarati only) |
| Page status | unknown |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 5010 |
| Publisher | Delhi Technological University, University of Utah, and Microsoft Research |
IndicXNLI is three-way natural language inference in eleven Indic languages. Given a premise and a hypothesis, the model must choose entailment, neutral, or contradiction. Aggarwal, Gupta and Kunchukuttan built the set by running IndicTrans over English XNLI, then checking translation quality with paired human ratings on a 100-sentence DPP sample of each language's test set. The paper lists Assamese, Gujarati (written "Gujarat ('gu')"), Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, Telugu, Hindi and Bengali. lm-evaluation-harness currently ships only Gujarati, because facebook/xnli already covers Hindi and does not include Gujarati.
Three-way classification. lm-eval task indicxnli_gu scores multiple-choice log-likelihood over a Gujarati cloze: premise + ", સાચું? {હા|તેથી|ના}, " + hypothesis, with gold label 0/1/2 mapped onto those three continuations. The YAML sets training_split train, validation_split validation and test_split test, so the default run uses the 5,010-row Gujarati test split.
No model card in ModelSpec reports this benchmark yet.