Nine-task Basque NLU suite in the GLUE mould, spanning NER, dialogue, topic, sentiment, stance, QNLI, word-in-context and coreference.
unassessed
| Category | composite |
|---|---|
| Subcategory | nine-task Basque NLU suite (NER, dialogue, topic, sentiment, stance, QNLI, WiC, coreference) |
| Page status | active |
| Metric | unweighted average of per-task scores (F1 or accuracy by task) |
| Direction | higher_is_better |
| Unit | % |
| Dataset licence | mixed: most tasks CC BY-NC-SA 4.0; QNLIeu CC BY-SA 4.0; VaxxStance CC BY 4.0 plus Twitter terms; BEC2016eu Twitter terms plus CC BY-NC-SA 4.0 |
| Publisher | orai NLP Technologies (Elhuyar) and HiTZ Center - Ixa, University of the Basque Country (UPV/EHU) |
BasqueGLUE scores Basque language understanding across nine tasks built from existing and newly adapted datasets, following GLUE and SuperGLUE design rules. Tasks include in- and out-of-domain NER, FMTOD intent classification and slot filling, news topic classification (BHTCv2), election-tweet sentiment (BEC2016eu), vaccine-stance detection, QNLI-style QA entailment, word-in-context, and binary coreference. The original evaluation fine-tunes encoder models per task and averages the nine scores.
Mixed: token-level sequence labelling (NER, slots); single-text classification (topic, sentiment, stance, intent); sentence-pair classification (QNLI, WiC, coreference). lm-eval implements six of the nine as Basque-prompted multiple-choice tasks and does not include NER or FMTOD.
No model card in ModelSpec reports this benchmark yet.