{
 "body": "\n## What it measures\n\nXNLIeu is three-way natural language inference in Basque. The model reads a Basque premise and hypothesis and chooses entailment, contradiction or neutral. The authors started from English [XNLI](xnli.md), machine-translated it, then had professionals post-edit the MT. They also wrote a native Basque test set from Basque news premises and speaker-written hypotheses, so they could see whether translation artefacts change which transfer method looks best.\n\nThe Hugging Face card's `pretty_name` field says \"Euska\u00f1olDS\"; that is not the paper's name. This page documents XNLIeu.\n\n## How it is scored\n\nEach pair is a three-way class, so chance is 33.3%. lm-evaluation-harness uses a Basque cloze \u2014 premise + \", ezta? Bai/Gainera/Ez, \" + hypothesis \u2014 and scores accuracy from sequence likelihood, matching the original XNLI harness recipe. Three task ids exist. `xnli_eu` is the postedited test (5,010 items). `xnli_eu_mt` is the raw MT test. `xnli_eu_native` is the 621-item native test. Unlike the original `xnli` group, these YAMLs set `test_split: test`.\n\nThe paper's own tables compare fine-tuned translate-train, zero-shot and typology setups as well as decoder prompting. Those protocols are not interchangeable with the harness cloze.\n\n## Dataset and licence\n\nPostedited `eu` and raw `eu_mt` each list 392,702 train, 2,490 validation and 5,010 test on Hugging Face; both configs reuse the MT train file. Native `eu_native` is 621 test pairs only. The paper scraped 5,000 Basque news sentences, kept 207 premises, and wrote three hypotheses per premise. Labels are public.\n\nThe GitHub README says the data follows the same licence as XNLI. The Hugging Face card tags CC BY-NC 4.0, matching the XNLI GitHub LICENSE.\n\n## Who publishes it\n\nMaite Heredia, Julen Etxaniz, Muitze Zulaika, Xabier Saralegi, Jeremy Barnes and Aitor Soroa, at the HiTZ Center / IXA, University of the Basque Country. Posted as arXiv:2404.06996 on 10 April 2024 and published at NAACL 2024 (long papers). Code and data: github.com/hitz-zentroa/xnli-eu.\n\n## Lineage\n\nPredecessor: [xnli](xnli.md), the 15-language 2018 eval this work extends. [basque_bench](basque_bench.md) includes XNLIeu as one Basque NLU constituent; a BasqueBench average is not an XNLIeu score. Do not treat `xnli_eu` as a sixteenth language inside the original `xnli` group.\n\n## Saturation and contamination\n\nNo current leaderboard ceiling was confirmed. The paper's result is that post-edition matters and that translate-train usually beats zero-shot, with a smaller gap on the native test. Contamination risk is high: postedited, MT and native labels are public, and the postedited set is a translation of already-public XNLI.\n\n## How to run it\n\nlm-eval: `xnli_eu`, `xnli_eu_mt`, or `xnli_eu_native` on `HiTZ/xnli-eu`. Default postedited eval is the 5,010-item test split. OpenCompass, inspect_evals, HELM and BIG-bench implementations were not found at the paths checked.\n\nName which of the three configs was used. Postedited and raw-MT accuracy are not the same number.\n\n## Reading the numbers\n\nA high postedited `xnli_eu` score means the model can do three-way Basque NLI on professionally cleaned translations of English XNLI. It does not by itself prove the model handles natively written Basque news inference; that is the 621-item native split, where the paper found smaller gains from translate-train. Do not mix this id with original `xnli` language tasks or with the [basque_bench](basque_bench.md) aggregate.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "xnli_eu",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "xnli-eu",
   "HiTZ/xnli-eu",
   "XNLIeu"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "Postedited and MT test labels are public on Hugging Face and GitHub. The postedited set is a translation of public [XNLI](xnli.md). The native 621 is also released with labels. No private hold-out is described.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "eu"
   ],
   "license": "CC BY-NC 4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 5010,
   "size_note": "Hugging Face `HiTZ/xnli-eu` and the dataset card: postedited `eu` and raw `eu_mt` each have 392,702 train / 2,490 validation / 5,010 test (the train file is the MT train in both configs). Native `eu_native` is test-only, 621 pairs. The headline `xnli_eu` harness task evaluates the postedited 5,010-item test split. The paper scraped 5,000 Basque news sentences, kept 207 as premises, and wrote three hypotheses each (621 pairs). 621 is the released native test count, not 5,000. The card's pretty_name is \"Euska\u00f1olDS\"; the paper name is XNLIeu.\n",
   "splits": "eu and eu_mt: train / validation / test; eu_native: test only (621)",
   "url": "https://huggingface.co/datasets/HiTZ/xnli-eu"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-016 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-016"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "xnli_eu",
   "opencompass": "",
   "other": "Also `xnli_eu_mt` and `xnli_eu_native`. Dataset `HiTZ/xnli-eu`. Unlike the original `xnli` group, `xnli_eu.yaml` sets test_split: test (5,010 rows). BasqueBench in this repository includes XNLIeu as one constituent of a larger Basque suite; that aggregate is not this page.\n"
  },
  "id": "xnli_eu",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "xnli",
   "successors": [],
   "variants": []
  },
  "measures": "XNLIeu extends [XNLI](xnli.md) to Basque. The model still classifies a premise\u2013hypothesis pair as entailment, contradiction or neutral, but the text is Basque. The authors first machine-translated English XNLI, then professionally post-edited that MT, and also built a smaller native Basque test set from scratch so they could check whether MT artefacts change which transfer recipe looks best.\n",
  "metric": {
   "baseline_note": "Three labels, so uniform chance is 33.3%. No pooled human-accuracy ceiling was confirmed from the sources opened here. The paper's claim is that post-edition changes measured transfer, not a human scoreboard.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 33.3,
   "unit": "%"
  },
  "name": "XNLIeu",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2404.06996",
   "title": "XNLIeu: a dataset for cross-lingual NLI in Basque",
   "url": "https://arxiv.org/abs/2404.06996",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Maite Heredia",
    "Julen Etxaniz",
    "Muitze Zulaika",
    "Xabier Saralegi",
    "Jeremy Barnes",
    "Aitor Soroa"
   ],
   "org": "HiTZ Center - IXA, University of the Basque Country (UPV/EHU)",
   "url": "https://github.com/hitz-zentroa/xnli-eu"
  },
  "released": "2024-04",
  "repo_url": "https://github.com/hitz-zentroa/xnli-eu",
  "saturation": {
   "as_of": "",
   "note": "No dedicated current leaderboard top score was confirmed. The paper compares post-edition versus raw MT and translate-train versus zero-shot on mono- and multilingual models rather than ranking frontier LLMs.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "XNLIeu: a dataset for cross-lingual NLI in Basque (arXiv:2404.06996)",
    "url": "https://arxiv.org/abs/2404.06996"
   },
   {
    "accessed": "2026-09-08",
    "title": "XNLIeu paper HTML (post-edition, native test, 5,000 news-sentence pool)",
    "url": "https://ar5iv.labs.arxiv.org/html/2404.06996"
   },
   {
    "accessed": "2026-09-08",
    "title": "NAACL 2024 XNLIeu anthology page",
    "url": "https://aclanthology.org/2024.naacl-long.234/"
   },
   {
    "accessed": "2026-09-08",
    "title": "hitz-zentroa/xnli-eu README (licence follows XNLI, three splits)",
    "url": "https://raw.githubusercontent.com/hitz-zentroa/xnli-eu/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "HiTZ/xnli-eu card (CC BY-NC 4.0, split table 5010 / 621)",
    "url": "https://huggingface.co/datasets/HiTZ/xnli-eu/raw/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "HiTZ/xnli-eu datasets-server sizes",
    "url": "https://datasets-server.huggingface.co/size?dataset=HiTZ/xnli-eu"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval xnli_eu README (three tasks)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/xnli_eu/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval xnli_eu.yaml (HiTZ/xnli-eu, test split, Basque cloze)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/xnli_eu/xnli_eu.yaml"
   }
  ],
  "status": "active",
  "subcategory": "Basque three-way NLI (postedited MT, raw MT, and native test)",
  "summary": "Basque XNLI: postedited and machine-translated English XNLI plus a 621-item native Basque test set, scored as three-way NLI accuracy.",
  "tags": [
   "nli",
   "basque",
   "cross-lingual",
   "entailment",
   "classification"
  ],
  "task_format": "Three-way classification. lm-evaluation-harness uses the same cloze as XNLI: premise + \", ezta? {Bai|Gainera|Ez}, \" + hypothesis, scored by multiple-choice log-likelihood. Three runnable tasks: `xnli_eu` (postedited, Hugging Face config `eu`), `xnli_eu_mt` (raw MT, `eu_mt`), `xnli_eu_native` (native test only, `eu_native`).\n"
 }
}