4,756 three-choice trivia questions, parallel in Basque and English and split almost evenly between Basque-local and general-global topics, to isolate what a model knows about one specific culture.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | parallel Basque/English trivia testing local (Basque-culture) versus global knowledge |
| Page status | active |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 4756 |
| Dataset licence | Stated two ways: the Hugging Face card's own metadata tags give CC BY 4.0, while the same card's free-text "License" field literally reads "[More Information Needed]"; this page records both rather than picking one. |
| Publisher | HiTZ Center - Ixa, University of the Basque Country (UPV/EHU) |
BertaQA is a multiple-choice trivia dataset built specifically to separate two kinds of knowledge that most benchmarks conflate: general "global" world knowledge, and "local" knowledge specific to one under-represented culture, here the Basque Country. Every question exists in parallel Basque and English versions, with the English version produced by professional human translation (plus several machine-translated variants used for ablation), so the same 4,756 questions can be asked in either language. The dataset's own release splits questions almost evenly into a "global" subset (2,392 questions, general-interest trivia) and a "local" subset (2,364 questions, requiring specific knowledge of Basque history, culture and society) -- confirmed directly by reading the released data's own `group` field rather than the paper's rounded description. This local/global split is the entire point of the benchmark: the authors use it to show that state-of-the-art LLMs do well on global topics but struggle specifically on local ones, and that continued pre-training in Basque measurably improves a model's local-topic performance even when the model is later queried in English -- evidence, the authors argue, of knowledge transfer from a low-resource to a high-resource language.
Three-option multiple-choice trivia (one correct answer, two distractors), evaluated zero-shot and few-shot. Each question also carries a category (one of eight) and a difficulty label. Besides the human-translated Basque (`eu`) and English (`en`) configs, the release ships eight further `en_mt_*` configs, the same questions machine-translated into English by different systems and models (NLLB, MADLAD, Latxa at several sizes, Llama 2 at several sizes, Gemma-7B), used to separate translation-quality effects from knowledge effects.
No model card in ModelSpec reports this benchmark yet.