XNLIeu

Basque XNLI: postedited and machine-translated English XNLI plus a 621-item native Basque test set, scored as three-way NLI accuracy.

Also known as: xnli-eu, HiTZ/xnli-eu, XNLIeu

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	reasoning
Subcategory	Basque three-way NLI (postedited MT, raw MT, and native test)
Page status	active
Metric	accuracy
Direction	higher_is_better
Unit	%
Dataset size	5010
Dataset licence	CC BY-NC 4.0
Publisher	HiTZ Center - IXA, University of the Basque Country (UPV/EHU)

What it measures

XNLIeu extends [XNLI](xnli.md) to Basque. The model still classifies a premise–hypothesis pair as entailment, contradiction or neutral, but the text is Basque. The authors first machine-translated English XNLI, then professionally post-edited that MT, and also built a smaller native Basque test set from scratch so they could check whether MT artefacts change which transfer recipe looks best.

Task format

Three-way classification. lm-evaluation-harness uses the same cloze as XNLI: premise + ", ezta? {Bai|Gainera|Ez}, " + hypothesis, scored by multiple-choice log-likelihood. Three runnable tasks: `xnli_eu` (postedited, Hugging Face config `eu`), `xnli_eu_mt` (raw MT, `eu_mt`), `xnli_eu_native` (native test only, `eu_native`).

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub