{ "body": "\n## What it measures\n\nBasqueGLUE is a Basque NLU suite, not a single skill test. A model is scored on nine tasks that together cover named-entity recognition, task-oriented dialogue (intent and slots), news-topic classification, tweet sentiment, vaccine stance, question-answer entailment, word-sense in context, and mention coreference. The LREC 2022 paper says the authors followed GLUE and SuperGLUE selection rules and reused Basque datasets where they could, adding new splits where they could not. Language is Basque (`eu`) throughout.\n\nlm-eval's `basqueglue` directory is a decoder-oriented slice of that suite: six multiple-choice tasks with Basque prompts. It drops NER and the FMTOD dialogue pair. A number from `--tasks basque-glue` is not the paper's nine-task encoder average.\n\n## How it is scored\n\nThe reference script `eval_basqueglue.py` compares a prediction file to each task's `test.jsonl`. Headline metric in the paper is an unweighted average of the nine task scores. NER contributes the mean of in-domain and out-of-domain F1. VaxxStance uses macro-F1 on FAVOR and AGAINST only. QNLI, WiC and coreference use accuracy. The 2022 baselines fine-tune BERTeus and ElhBERTeu separately on each task.\n\nlm-eval scores the six implemented tasks as multiple-choice, with micro-F1 on BHTC and BEC, a two-class F1 on VaxxStance, and accuracy on QNLI, WiC and coreference. The tag `basque-glue` runs all six and does not average them. Do not drop a single \"BasqueGLUE\" percentage from lm-eval without saying which tasks went into it.\n\n## Dataset and licence\n\nThere is no one row count. Table 1 of the paper (copied on GitHub and Hugging Face) lists train/val/test sizes per task. NER and slot rows are tokens; the rest are examples. Test data are public. Licences differ by task: most are CC BY-NC-SA 4.0, QNLIeu is CC BY-SA 4.0, VaxxStance is CC BY 4.0 plus Twitter's terms, and BEC2016eu adds Twitter's terms to CC BY-NC-SA 4.0. The evaluation script itself is CC BY-SA 4.0. The LREC PDF is under ELRA CC-BY-NC-4.0, which is the paper, not the data.\n\n## Who publishes it\n\nGorka Urbizu, I\u00f1aki San Vicente and Xabier Saralegi (then Elhuyar / now orai NLP) with Rodrigo Agerri and Aitor Soroa (HiTZ Center - Ixa, UPV/EHU). The paper is LREC 2022, Marseille, 20\u201325 June 2022, pages 1603\u20131612. Current repo is `orai-nlp/BasqueGLUE`; the PDF still mentions `Elhuyar/BasqueGLUE`. Hugging Face: `orai-nlp/basqueGLUE`, last updated 2024-04-08.\n\n## Lineage\n\nEnglish [GLUE](glue.md) and SuperGLUE are the design templates. [BasqueBench](basque_bench.md) later reuses BasqueGLUE's QNLIeu inside the IberoBench Basque slice, alongside many tasks BasqueGLUE does not contain. Latxa (arXiv:2403.20266) uses BasqueGLUE to score decoder models. This page is the original nine-task suite, not BasqueBench.\n\n## Saturation and contamination\n\nA 74% fine-tuned BERT average from 2022 does not tell you whether today's Basque LLMs have used up the suite. Decoder numbers need the Latxa paper or a fresh lm-eval run. Test labels are public, and several source domains are old, so contamination is a live concern, especially for Wikipedia QNLI and news NER.\n\n## How to run it\n\nOfficial: fine-tune, write `test.jsonl`-shaped predictions, run `python3 eval_basqueglue.py --task ...`. lm-eval: `--tasks basque-glue` or the six names `bhtc_v2`, `bec2016eu`, `vaxx_stance`, `qnlieu`, `wiceu`, `epec_koref_bin`, dataset `orai-nlp/basqueGLUE`. WiC decoding in lm-eval re-encodes strings as latin-1 then utf-8 to repair an encoding issue. Prompt language is Basque. HELM and OpenCompass names were not found.\n\n## Reading the numbers\n\nA 73% \"BasqueGLUE\" figure from 2022 is an encoder fine-tune average, not a zero-shot LLM score. An lm-eval run on six tasks is a different slice and a different protocol. Check which tasks, which metric, and whether NER was included. Pair it with [BasqueBench](basque_bench.md) if you care about exam QA and translation as well as this older NLU core. Non-commercial licences on most tasks also limit redistribution even when the average looks open.\n", "build": { "built_at": "2026-09-09T16:56:50+00:00", "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11", "eligibility_as_of": "2026-09-09" }, "disposition": { "canonical_id": "basqueglue", "reasons": [], "status": "unassessed", "verified_results": [] }, "models_covered": [], "page": { "aliases": [ "Basque GLUE", "orai-nlp/basqueGLUE" ], "category": "composite", "contamination": { "note": "Test sets are public, as SuperGLUE-style design required. Several source corpora (news, Wikipedia, tweets, Wordnet) predate 2022 and may appear in Basque pretraining. No rotating held-out set is described. Tweet tasks also carry Twitter terms of use.\n", "risk": "medium" }, "dataset": { "languages": [ "eu" ], "license": "mixed: most tasks CC BY-NC-SA 4.0; QNLIeu CC BY-SA 4.0; VaxxStance CC BY 4.0 plus Twitter terms; BEC2016eu Twitter terms plus CC BY-NC-SA 4.0", "modalities": [ "text" ], "public_test_set": true, "size": null, "size_note": "Not one item pool. Paper Table 1 (and the GitHub/HF README) lists per-task train/val/test sizes. Sequence-labelling rows are token counts (NERC, slots); the others are example counts. WiCeu train alone is 408,559 pairs, which drives the Hugging Face 100K