{
 "body": "\n## What it measures\n\nGalicianBench is the Galician-language slice of IberoBench, the multilingual, multi-task project that\nalso produced this repository's [CatalanBench](catalan_bench.md) and [BasqueBench](basque_bench.md)\npages, one same-shaped suite per official language of the Iberian peninsula. It bundles 11 top-level\nlm-evaluation-harness task groups, one of which (flores_gl) itself expands into 16 directional\ntranslation subtasks. Provenance splits cleanly across the suite, confirmed dataset by dataset rather\nthan assumed: GalCoLA (linguistic acceptability, 17,088 sentences drawn from PROPOR 2022 and ACL 2023\nGalician syntax research), summarization_gl (80,829 native news-article/summary pairs from three\nGalician outlets -- Nos Diario, Que Pasa na Costa, Praza Publica) and parafrases_gl (2,032\nthree-way-annotated paraphrase pairs, sourced from native Galician Wikipedia, novels and parliamentary\nsessions, with paraphrase variants machine-generated by term replacement and back-translation, then\nmanually reviewed by two linguists) were built directly in Galician. By contrast, openbookqa_gl, mgsm_direct_gl, xstorycloze_gl, truthfulqa_gl, xnli_gl and\npaws_gl are explicit translations of their English originals (OpenBookQA, MGSM, StoryCloze, TruthfulQA,\nXNLI, PAWS respectively, each dataset card states this directly), and belebele_glg_Latn and flores_gl\nare Galician configurations of already-multilingual, professionally translated suites.\n\n## How it is scored\n\nScoring is task-dependent, the norm for a composite suite this broad: multiple-choice and\nnarrative-completion tasks (openbookqa_gl, xstorycloze_gl, belebele_glg_Latn) score by accuracy;\nclassification tasks (galcola, parafrases_gl, paws_gl, xnli_gl) score by accuracy or F1 over two or\nthree label classes; mgsm_direct_gl scores free-form numeric exact match; summarization_gl and\ntruthfulqa_gl's generation split score by ROUGE or similar text-overlap metrics; and flores_gl's 16\ntranslation directions score on continuous machine-translation metrics. Because of this mix, the\nIberoBench paper reports its own headline numbers as a Normalized Preferred Metric (NPM), rescaling\neach task so a random guess maps to 0 and the maximum possible score maps to 100, evaluated under both\n0-shot and 5-shot prompting across 33 base models. The paper's own per-language aggregation table uses\na fixed 14-task subset for Galician specifically, not the full 11-group-plus-flores-directions harness\nlisting.\n\n## Dataset and licence\n\nGalicianBench has no single dataset size, since it aggregates independently sized, independently\nsourced datasets. Confirmed component sizes: GalCoLA 17,088 sentences, summarization_gl 80,829\nnews/summary pairs, parafrases_gl 2,032 sentence pairs, openbookqa_gl 500 validation + 500 test, mgsm_gl\n8 train + 250 test, and xstorycloze_gl 360 train + 1,511 test. The IberoBench paper's own Table 7 names\nexactly 14 tasks it used for Galician's aggregate score, read directly for this page rather than taken\non trust: belebele_glg_Latn, flores_gl, galcola, mgsm_direct_gl, openbookqa_gl, parafrases_gl, paws_gl,\nsummarization_gl, three truthfulqa_gl sub-metrics, and three veritasqa_gl sub-metrics -- the latter\nbelonging to VeritasQA, a second Galician truthfulness dataset (Aula-Blasco et al. 2025) this page\ncould not locate in the harness as currently published (see Lineage). Licensing is mostly but not\nentirely uniform: CC BY 4.0 covers galcola, summarization_gl, parafrases_gl, PAWS-gl, openbookqa_gl,\nmgsm_gl, xstorycloze_gl and belebele_gl (each confirmed individually), while truthfulqa_gl is\nApache-2.0 and xnli_gl is CC BY-NC 4.0 -- meaning no single licence covers GalicianBench as a whole.\n\n## Who publishes it\n\nGalicianBench's Galician-specific datasets are published under Proxecto Nos, the Galician-language AI\ninitiative coordinated through the Xunta de Galicia and the CiTIUS research centre at the Universidade\nde Santiago de Compostela. The suite sits within the wider IberoBench project, published by Irene\nBaucells and fourteen co-authors at COLING 2025 (Abu Dhabi, January 2025) -- the same paper and author\ngroup behind CatalanBench and BasqueBench. The task group itself is maintained inside EleutherAI's\nlm-evaluation-harness, with a changelog showing an update as recent as January 2026.\n\n## Lineage\n\nGalicianBench is one of five same-shaped IberoBench suites, sharing its construction approach and NPM\nscoring with [CatalanBench](catalan_bench.md) and [BasqueBench](basque_bench.md), both catalogued in\nthis repository, plus PortugueseBench and SpanishBench, which are not. A reader comparing GalicianBench\nscores against CatalanBench or BasqueBench scores is comparing siblings built by the same team under\nthe same methodology, not unrelated projects. Within GalicianBench itself, note that parafrases_gl and\nPAWS-gl are two distinct paraphrase-detection tasks rather than duplicates: parafrases_gl starts from\nnative Galician source text with paraphrase variants then generated and manually reviewed, using\nthree-way labels, while PAWS-gl is a direct translation of the English PAWS dataset with binary labels\n-- broadly the same native/translated split CatalanBench draws between its own Parafraseja and PAWS-ca\ntasks. GalicianBench has no other predecessor or successor tracked here.\nOne open discrepancy: the IberoBench paper's own Table 7 used a second Galician truthfulness dataset,\nVeritasQA_gl (Aula-Blasco et al. 2025), alongside truthfulqa_gl for its published Galician score, but\nthis page could not find veritasqa_gl in the lm-evaluation-harness galician_bench group as currently\npublished (checked directly in both the README and the YAML) -- either it was removed, renamed, or\nnever merged from whatever snapshot the paper used; this page reports the gap rather than guessing\nwhich.\n\n## Saturation and contamination\n\nNo maintained leaderboard specific to GalicianBench was found, so saturation is recorded as \"open\"\nrather than \"saturated,\" on the strength of the shared IberoBench paper's finding that \"model\nperformance in Iberian languages still is behind state-of-the-art results\" across the 33 base models it\nevaluated. The paper's own Table 2 shows this acutely for Galician: the best Galician-specific model,\nCarballo-B-1.3B, scores 2.70 (normalized), far below Llama 3-8B's 18.59 -- a general multilingual model\nwith no Galician-specific training at all outperforming every Galician-tuned model in the comparison.\nContamination risk is medium and uneven: the English or multilingual originals behind\nGalicianBench's translated components (XNLI, PAWS, Belebele, FLORES, OpenBookQA, StoryCloze,\nTruthfulQA) have been public for years, while the natively Galician components and the Galician\ntranslations themselves are newer and specific to this project, with no publisher statement or\nindependent contamination study covering the suite as a whole.\n\n## How to run it\n\nlm-evaluation-harness implements GalicianBench as the `galician_bench` group, aggregating 11\nsub-task/group YAML files (for example `galcola`, `openbookqa_gl`, `xnli_gl`, `flores_gl`); a\nchangelog entry dated 2026-01-16 (v1.1) excluded line breaks from stop criteria in `mgsm_direct_gl`.\nThe README's own task table mislabels the math task's homepage row \"MGSM_ca,\" an apparent\ncopy-paste artifact from the CatalanBench README rather than a change to what the task measures. No\nHELM, OpenCompass or inspect_evals implementation of the group was found. Because the suite mixes\naccuracy-, F1-, ROUGE- and translation-metric-scored tasks, an aggregate \"GalicianBench score\" is not\ndirectly comparable to a single-metric benchmark; compare at the level of individual tasks or\ncategories, matching how the IberoBench paper itself reports results.\n\n## Reading the numbers\n\nGalicianBench is best read task by task rather than as one blended score, given how many skills and\nmetrics it bundles. Because three of its eleven components were purpose-built in Galician while the\nrest are translations, a strong overall showing does not by itself say whether a model is strong on\ngenuinely native Galician content or mainly on translated material -- check the native components\n(GalCoLA, summarization_gl, parafrases_gl) separately if that distinction matters for your use case.\nAs with its CatalanBench and BasqueBench siblings, no combined public leaderboard exists, so treat any\nsingle reported \"GalicianBench\" number as one reporter's own aggregation choice and check which\nsub-tasks and shot-count it used before comparing it to another.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "galician_bench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "composite",
  "contamination": {
   "note": "Exposure varies by component: translated sets built on long-established multilingual resources (XNLI, PAWS, Belebele, FLORES, OpenBookQA, StoryCloze, TruthfulQA) have underlying English or multilingual originals that have been public for years, while the natively Galician components (GalCoLA, summarization_gl, parafrases_gl) and the Galician translations themselves are newer and specific to this project. No publisher statement or independent contamination study covering the suite as a whole was found.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "gl"
   ],
   "license": "Mostly CC BY 4.0 (galcola, summarization_gl, parafrases_gl, PAWS-gl, openbookqa_gl, mgsm_gl, xstorycloze_gl and belebele_gl, confirmed individually via the Hugging Face API), with two exceptions: truthfulqa_gl is Apache-2.0 and xnli_gl is CC BY-NC 4.0. This page did not check every component individually, so treat any single suite-wide licence claim with caution.\n",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "GalicianBench aggregates independently sized source datasets rather than one fixed item pool. Sizes confirmed directly from component dataset cards: GalCoLA 17,088 sentences, summarization_gl 80,829 news/summary pairs (the paper's own text rounds this to \"over 80,000\"), parafrases_gl 2,032 sentence pairs, openbookqa_gl 500 validation + 500 test, mgsm_gl 8 train + 250 test, and xstorycloze_gl 360 train + 1,511 test. The IberoBench paper's own Table 7 (read directly, in full, for this page) lists exactly 14 tasks used to compute its Galician aggregate score: belebele_glg_Latn, flores_gl, galcola, mgsm_direct_gl, openbookqa_gl, parafrases_gl, paws_gl, summarization_gl, truthfulqa_gl_gen, truthfulqa_gl_mc1, truthfulqa_gl_mc2, veritasqa_gen_gl, veritasqa_mc1_gl and veritasqa_mc2_gl (against 27 for Catalan, 17 for Spanish, 14 for Basque and 4 for Portuguese). That list includes three veritasqa_gl sub-metrics this page could not find in the current lm-evaluation-harness galician_bench.yaml or README (see Lineage) -- a second truthfulness dataset, VeritasQA (Aula-Blasco et al. 2025), that the paper's own scoring used but that appears absent from, or renamed in, the harness as currently published. The same table also omits xnli_gl and xstorycloze_gl, both present in the current 11-entry harness group -- the task set has evidently changed on both ends since the paper's own snapshot. lm-evaluation-harness's galician_bench.yaml itself lists 11 top-level entries, one of which (flores_gl) expands into 16 directional translation subtasks on its own.\n",
   "splits": "aggregates each source dataset's own splits; most components ship train/validation/test or train/test splits documented on their individual Hugging Face cards",
   "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/galician_bench"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 6, slice C",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "galician_bench (group); constituent tasks/groups: belebele_glg_Latn, flores_gl (plus 16 directional variants), galcola, summarization_gl, parafrases_gl, paws_gl, openbookqa_gl, mgsm_direct_gl, truthfulqa_gl, xnli_gl, xstorycloze_gl",
   "opencompass": "",
   "other": "lm-evaluation-harness's galician_bench.yaml lists 11 top-level entries; its own changelog records a v1.1 update on 2026-01-16 that excluded line breaks from stop criteria in mgsm_direct_gl. The README's task table mistakenly labels the math task's homepage row \"MGSM_ca\" rather than \"MGSM_gl\" -- an apparent copy-paste carry-over from the CatalanBench README rather than a change in what the task actually evaluates.\n"
  },
  "id": "galician_bench",
  "last_updated": "2026-01",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "GalicianBench is the Galician-language slice of IberoBench, the same project behind this repository's CatalanBench and BasqueBench pages, covering the official languages of the Iberian peninsula. It bundles 11 top-level lm-evaluation-harness task groups (one of which, flores_gl, itself expands into 16 directional translation subtasks). Unlike some sibling suites, provenance splits cleanly and was confirmed dataset by dataset: GalCoLA (linguistic acceptability, 17,088 sentences), summarization_gl (80,829 native news-article/summary pairs from three Galician outlets) and parafrases_gl (2,032 sentence pairs, sourced from Galician Wikipedia, novels and parliamentary sessions, with paraphrase variants generated by term replacement and back-translation then manually reviewed by two linguists, per the IberoBench paper's own dataset description) were built directly in Galician; openbookqa_gl, mgsm_direct_gl, xstorycloze_gl, truthfulqa_gl, xnli_gl and paws_gl are explicit translations of their English originals (OpenBookQA, MGSM, StoryCloze, TruthfulQA, XNLI, PAWS); and belebele_glg_Latn and flores_gl are Galician configurations of professionally translated multilingual suites (Belebele, FLORES).\n",
  "metric": {
   "baseline_note": "No single random-guess figure applies across a suite this heterogeneous: multiple-choice sub-tasks vary in option count, translation sub-tasks are scored on continuous metrics rather than accuracy, and the IberoBench paper reports its own comparisons using a Normalized Preferred Metric (NPM) that rescales each task so a random guess maps to 0 and the maximum possible score maps to 100 -- the mechanism that lets the paper average differently-scaled tasks into one per-language figure. No human baseline is reported.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "task-dependent: accuracy for most multiple-choice, NLI and classification tasks; ROUGE for summarisation; BLEU/ChrF-family scores for flores_gl",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "GalicianBench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "IberoBench: A Benchmark for LLM Evaluation in Iberian Languages",
   "url": "https://aclanthology.org/2025.coling-main.699",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Irene Baucells",
    "Javier Aula-Blasco",
    "Iria de-Dios-Flores",
    "Silvia Paniagua Su\u00e1rez",
    "Naiara Perez",
    "Anna Salles",
    "Susana Sotelo Docio",
    "J\u00falia Falc\u00e3o",
    "Jose Javier Saiz",
    "Robiert Sepulveda Torres",
    "Jeremy Barnes",
    "Pablo Gamallo",
    "Aitor Gonzalez-Agirre",
    "German Rigau",
    "Marta Villegas"
   ],
   "org": "Proxecto Nos (Galician-language AI initiative run through the Xunta de Galicia and the CiTIUS research centre at the Universidade de Santiago de Compostela), within the IberoBench project led by the Barcelona Supercomputing Center (BSC-CNS)",
   "url": "https://huggingface.co/proxectonos"
  },
  "released": "2025-01",
  "repo_url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/galician_bench",
  "saturation": {
   "as_of": "2025-01",
   "note": "No maintained leaderboard specific to GalicianBench was found. The shared IberoBench paper evaluated 33 base models under 0-shot and 5-shot prompting and states that \"model performance in Iberian languages still is behind state-of-the-art results.\" Its own Table 2 (5-shot, normalized scores) shows this holds in an unusually stark way for Galician specifically: the best Galician-specific model, Carballo-B-1.3B, scores 2.70, far below the best-scoring model overall for that column, Llama 3-8B, at 18.59 -- a general multilingual model with no Galician-specific training, read directly from the paper's own results table. top_score is left unset here since a single normalized figure from one specific paper's methodology is not the same claim as a leaderboard ceiling.\n",
   "status": "open",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness galician_bench task group README",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/galician_bench/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness galician_bench.yaml (11-entry group definition)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/galician_bench/galician_bench.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "Baucells et al. (2025). IberoBench: A Benchmark for LLM Evaluation in Iberian Languages. COLING 2025, full PDF",
    "url": "https://aclanthology.org/2025.coling-main.699.pdf"
   },
   {
    "accessed": "2026-09-08",
    "title": "proxectonos/galcola dataset card (17,088 sentences, PROPOR 2022 / ACL 2023 sources)",
    "url": "https://huggingface.co/datasets/proxectonos/galcola/raw/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "proxectonos/summarization_gl dataset card (80,829 native Galician news/summary pairs)",
    "url": "https://huggingface.co/datasets/proxectonos/summarization_gl/raw/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "proxectonos/parafrases_gl dataset card (native Galician text selection, 3-way annotation)",
    "url": "https://huggingface.co/datasets/proxectonos/parafrases_gl/raw/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "proxectonos/openbookqa_gl dataset card (translated from English OpenBookQA)",
    "url": "https://huggingface.co/datasets/proxectonos/openbookqa_gl/raw/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "proxectonos/mgsm_gl dataset card (translated from English MGSM)",
    "url": "https://huggingface.co/datasets/proxectonos/mgsm_gl/raw/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "proxectonos/xstorycloze_gl dataset card (translated from English StoryCloze)",
    "url": "https://huggingface.co/datasets/proxectonos/xstorycloze_gl/raw/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "proxectonos/truthfulqa_gl dataset card (Galician version of TruthfulQA)",
    "url": "https://huggingface.co/datasets/proxectonos/truthfulqa_gl/raw/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "proxectonos/xnli_gl dataset card (extension of XNLI, translated to Galician)",
    "url": "https://huggingface.co/datasets/proxectonos/xnli_gl/raw/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "proxectonos/PAWS-gl dataset card (translation of English PAWS)",
    "url": "https://huggingface.co/datasets/proxectonos/PAWS-gl/raw/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "proxectonos/belebele_gl dataset metadata, Hugging Face API (licence cc-by-sa-4.0)",
    "url": "https://huggingface.co/api/datasets/proxectonos/belebele_gl"
   }
  ],
  "status": "active",
  "subcategory": "Galician-language multitask suite: linguistic acceptability, QA, NLI, paraphrase, summarisation, commonsense reasoning, math and translation",
  "summary": "The IberoBench suite for Galician: 11 lm-evaluation-harness task groups mixing three natively built Galician resources with eight translated from English or multilingual sources.",
  "tags": [
   "composite",
   "galician",
   "iberobench",
   "multilingual",
   "low-resource",
   "nli",
   "translation",
   "question-answering"
  ],
  "task_format": "Mixed by sub-task: four-option multiple-choice for openbookqa_gl; binary acceptability classification for galcola; three-way paraphrase classification for parafrases_gl and binary for paws_gl; three-way natural-language-inference classification for xnli_gl; two-ending narrative completion for xstorycloze_gl; free-form generation for truthfulqa_gl's generation split and summarization_gl; free-form grade-school math word problems for mgsm_direct_gl; multiple-choice reading comprehension for belebele_glg_Latn; and bidirectional machine translation between Galician and eight other languages for flores_gl.\n"
 }
}