HEAD-QA scores multiple-choice questions from real Spanish healthcare civil-service exams, released in matched Spanish and English versions.
unassessed
| Category | domain |
|---|---|
| Subcategory | Spanish healthcare civil-service exam question answering, cross-lingual (Spanish/English) |
| Page status | active |
| Metric | accuracy (and length-normalized accuracy, acc_norm) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 6765 |
| Dataset licence | MIT, per the original aghie/head-qa GitHub repository's LICENSE file and the dvilares/head_qa dataset card; the separate EleutherAI/headqa mirror used by lm-evaluation-harness instead lists its own licence tag as "other" without further detail, so that specific mirror's terms should not be assumed to be MIT. |
| Publisher | Universidade da Coruña |
HEAD-QA tests whether a model can answer multiple-choice questions taken from real exams used to award specialized positions in the Spanish public healthcare system, set by Spain's Ministerio de Sanidad, Consumo y Bienestar Social. Questions cover six subjects -- medicine, nursing, psychology, chemistry, pharmacology and biology -- and were written for practicing or aspiring health professionals rather than for the benchmark, so they assume applied domain training rather than general knowledge. The dataset ships in two matched forms, the original Spanish and a professionally produced English translation, letting the same questions be evaluated monolingually or cross-lingually.
Multiple-choice question with a variable number of answer options; the model reads the question text and returns the correct option. Most items are plain text, but a minority carry an accompanying image, and text-only harnesses typically evaluate only the text subset.
No model card in ModelSpec reports this benchmark yet.