{
 "body": "\n## What it measures\n\nHEAD-QA tests whether a model can answer multiple-choice questions taken from real exams used to award\nspecialized positions in the Spanish public healthcare system, set by Spain's Ministerio de Sanidad,\nConsumo y Bienestar Social. Questions cover six subjects -- medicine, nursing, psychology, chemistry,\npharmacology and biology -- and were written for practicing or aspiring health professionals, not for the\nbenchmark, so they assume the kind of applied domain training a licensing exam expects rather than\ngeneral knowledge. The dataset ships in two matched forms: the original Spanish and a professionally\nproduced English translation, so the same questions can be scored monolingually or cross-lingually.\n\nMost questions are plain text, but a minority carry an accompanying image (a diagram, chart or clinical\nimage); text-only harnesses typically evaluate only the text subset, so a reported score usually excludes\nimage-bearing items unless the reporter says otherwise.\n\n## How it is scored\n\nModels are graded on accuracy: the share of questions answered with the correct option. Because the\nnumber of answer choices is not fixed at four across every question, the paper does not quote a single\nrandom-guessing baseline. lm-evaluation-harness additionally reports `acc_norm`, an accuracy variant that\nnormalizes each option's log-likelihood by its length before selecting the model's answer, common practice\nfor multiple-choice tasks whose options vary in token length. Evaluation is zero-shot by default in\nlm-evaluation-harness, whose task file sets no few-shot count; HELM's scenario implementation restricts\nscoring to the test split.\n\n## Dataset and licence\n\n6,765 questions per language, split 2,657 train / 1,366 validation / 2,742 test on the EleutherAI/headqa\nmirror that lm-evaluation-harness reads (the original dvilares/head_qa release carries the same split\nsizes). The six subject categories are medicine, nursing, psychology, chemistry, pharmacology and biology,\ndrawn from Spanish civil-service exams from the years leading up to the dataset's 2019 release. The\nauthors' GitHub repository carries an MIT LICENSE file, and the dvilares Hugging Face card also tags the\ndataset MIT; the separate EleutherAI/headqa mirror used by the harness instead lists its own licence as\n\"other\" without elaborating, so that mirror's terms should not be assumed to match. Test-split answers are\nincluded in the public files; there is no gating.\n\n## Who publishes it\n\nHEAD-QA was introduced by David Vilares and Carlos G\u00f3mez-Rodr\u00edguez, then at the Universidade da Coru\u00f1a, in\n\"HEAD-QA: A Healthcare Dataset for Complex Reasoning,\" presented at ACL 2019. The authors maintain the\nreference data and code at github.com/aghie/head-qa; the Hugging Face mirrors used by lm-evaluation-harness\nand HELM are maintained by those respective projects rather than by the original authors, and no dedicated\nleaderboard site was found for this page.\n\n## Lineage\n\nHEAD-QA has no formal predecessor, successor or variant catalogued in this repository. It sits in the same\nbroad space as MedQA and MedMCQA (both also in this repository) as a real-exam medical multiple-choice\nbenchmark, but was built independently on Spanish rather than US or Indian exam material, and ships a\nbuilt-in cross-lingual English translation that those two do not.\n\n## Saturation and contamination\n\nNo current leaderboard or recent model report was found during this research pass, so a present-day top\nscore cannot be stated with a source behind it. HEAD-QA remains a registered task in both EleutherAI's\nlm-evaluation-harness and Stanford HELM's scenario library, both actively maintained projects, but neither\nproject's own results page was opened here, and this repository's own model cards do not carry a headqa\nscore either. Contamination risk is high: the question-and-answer set, including test-split answers, has\nbeen publicly downloadable without gating since 2019, and the lm-evaluation-harness task config itself\nsets `should_decontaminate: true`, flagging HEAD-QA as text its maintainers consider worth checking against\ntraining corpora.\n\n## How to run it\n\nlm-evaluation-harness registers `headqa_en` and `headqa_es` under the group tag `headqa`, reading the\nEleutherAI/headqa mirror and reporting `acc` and `acc_norm` zero-shot. HELM's `headqa_scenario.py` reads\nthe dvilares/head_qa release directly, defaults to the text-only subset (skipping image-bearing questions\nunless configured otherwise), and scores only the test split. Because the two harnesses read different\nmirrors of the same underlying data and differ in whether they include image-bearing items, a score from\nlm-evaluation-harness is not guaranteed to be computed over an identical question set to a HELM score.\n\n## Reading the numbers\n\nA HEAD-QA score reflects a model's grasp of Spanish (or, in translation, English) healthcare\nlicensing-exam content across six clinical and biomedical subjects, not general medical reasoning in some\nother exam format -- compare it to MedQA or MedMCQA scores only loosely, since the source exams, languages\nand option counts all differ. Because the dataset has been public for years with test answers included,\nand because no fresh leaderboard reading was available for this page, treat a very high score with some\ncaution about training-data exposure. Check whether the reporter used the Spanish or English form and\nwhether image-bearing questions were included, since either choice changes what is actually being\nmeasured.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "headqa",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "HeadQA"
  ],
  "category": "domain",
  "contamination": {
   "note": "The question-and-answer set, including test-split answers, has been publicly downloadable without gating since 2019. The lm-evaluation-harness task config itself sets `should_decontaminate: true`, meaning its maintainers flag HEAD-QA's text as something a training-corpus decontamination check should look for.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "es",
    "en"
   ],
   "license": "MIT, per the original aghie/head-qa GitHub repository's LICENSE file and the dvilares/head_qa dataset card; the separate EleutherAI/headqa mirror used by lm-evaluation-harness instead lists its own licence tag as \"other\" without further detail, so that specific mirror's terms should not be assumed to be MIT.\n",
   "modalities": [
    "text",
    "image"
   ],
   "public_test_set": true,
   "size": 6765,
   "size_note": "6,765 questions per language (the Spanish original and its English translation cover the same questions), split 2,657 train / 1,366 validation / 2,742 test on the EleutherAI/headqa Hugging Face mirror that lm-evaluation-harness reads. Six subject categories: medicine, nursing, psychology, chemistry, pharmacology, biology, drawn from Spanish civil-service exams from the years leading up to the dataset's 2019 release. A separate images/PDFs archive accompanies some questions.\n",
   "splits": "2,657 train / 1,366 validation / 2,742 test (per language)",
   "url": "https://huggingface.co/datasets/dvilares/head_qa"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 2, slice F",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "headqa",
   "inspect_evals": "",
   "lm_eval": "headqa",
   "opencompass": "",
   "other": "lm-evaluation-harness registers `headqa_en` and `headqa_es` under the group tag `headqa`, reading the EleutherAI/headqa mirror and reporting `acc` and `acc_norm`, zero-shot by default (its task YAML sets no few-shot count). HELM's scenario class, `headqa_scenario.py`, reads dvilares/head_qa directly, defaults to the text-only subset, and restricts scoring to the test split.\n"
  },
  "id": "headqa",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "HEAD-QA tests whether a model can answer multiple-choice questions taken from real exams used to award specialized positions in the Spanish public healthcare system, set by Spain's Ministerio de Sanidad, Consumo y Bienestar Social. Questions cover six subjects -- medicine, nursing, psychology, chemistry, pharmacology and biology -- and were written for practicing or aspiring health professionals rather than for the benchmark, so they assume applied domain training rather than general knowledge. The dataset ships in two matched forms, the original Spanish and a professionally produced English translation, letting the same questions be evaluated monolingually or cross-lingually.\n",
  "metric": {
   "baseline_note": "The paper does not quote one numeric random baseline, since the number of options is not fixed at four across every question, nor a physician baseline score; it states only that model results \"lag well behind human performance.\"\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy (and length-normalized accuracy, acc_norm)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "HEAD-QA",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "1906.04701",
   "title": "HEAD-QA: A Healthcare Dataset for Complex Reasoning",
   "url": "https://arxiv.org/abs/1906.04701",
   "year": 2019
  },
  "publisher": {
   "authors": [
    "David Vilares",
    "Carlos G\u00f3mez-Rodr\u00edguez"
   ],
   "org": "Universidade da Coru\u00f1a",
   "url": "https://github.com/aghie/head-qa"
  },
  "released": "2019-06",
  "repo_url": "https://github.com/aghie/head-qa",
  "saturation": {
   "as_of": "",
   "note": "No current leaderboard or recent model report was found during this research pass, so a present-day top score cannot be stated with a source behind it. HEAD-QA remains a registered task in both EleutherAI's lm-evaluation-harness (`headqa_en`/`headqa_es`) and Stanford HELM's scenario library, both actively maintained projects, but neither project's own results page was opened here.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HEAD-QA: A Healthcare Dataset for Complex Reasoning",
    "url": "https://arxiv.org/abs/1906.04701"
   },
   {
    "accessed": "2026-09-08",
    "title": "dvilares/head_qa dataset card API, Hugging Face",
    "url": "https://huggingface.co/api/datasets/dvilares/head_qa"
   },
   {
    "accessed": "2026-09-08",
    "title": "EleutherAI/headqa dataset card API, Hugging Face",
    "url": "https://huggingface.co/api/datasets/EleutherAI/headqa"
   },
   {
    "accessed": "2026-09-08",
    "title": "EleutherAI/headqa split sizes, Hugging Face datasets-server",
    "url": "https://datasets-server.huggingface.co/size?dataset=EleutherAI/headqa"
   },
   {
    "accessed": "2026-09-08",
    "title": "aghie/head-qa repository",
    "url": "https://github.com/aghie/head-qa"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness: headqa_en task config",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/headqa/headqa_en.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness: headqa task README",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/headqa/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM headqa_scenario.py",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/headqa_scenario.py"
   }
  ],
  "status": "active",
  "subcategory": "Spanish healthcare civil-service exam question answering, cross-lingual (Spanish/English)",
  "summary": "HEAD-QA scores multiple-choice questions from real Spanish healthcare civil-service exams, released in matched Spanish and English versions.",
  "tags": [
   "medical",
   "multiple-choice",
   "spanish",
   "cross-lingual",
   "civil-service-exam"
  ],
  "task_format": "Multiple-choice question with a variable number of answer options; the model reads the question text and returns the correct option. Most items are plain text, but a minority carry an accompanying image, and text-only harnesses typically evaluate only the text subset.\n"
 }
}