{
 "body": "\n## What it measures\n\n`mediqa_qa2019` is lm-eval's generation wrap of MEDIQA 2019 Task 3. It is not MedQA, and it is not the 2019 ranking contest as scored. The shared task gave a consumer health question and CHiQA answers, then asked teams to drop bad answers and re-rank the rest. lm-eval keeps the question string and asks the model to write an answer. The language is English. The output is free text.\n\nThe 2019 workshop also ran medical NLI and recognizing-question-entailment tracks. Those sets and metrics are separate. HELM's [medi_qa](medi_qa.md) uses the same questions with a different gold and an LLM jury. Do not treat the two ids as one number.\n\n## How it is scored\n\nlm-eval `mediqa_qa2019` is `generate_until` with stop sequence `\\n\\n`. It reports BLEU, ROUGE-1/2/L, BLEURT (bleurt-base-512) and English BERTScore F1, each aggregated with nanmean. The YAML does not name a headline metric. Empty predictions or empty golds become NaN.\n\nGold is `AnswerList[0]` in the Hugging Face source schema. That is XML order in the loader, not the expert's `ReferenceRank=1` string that HELM uses. The 2019 ranking headline of 78.3% among 72 teams is a different scale. No random or human baseline was stated for this overlap protocol. `mediqa_qa2019_perplexity` instead reports rolling perplexity of the same gold string.\n\n## Dataset and licence\n\nHugging Face `bigbio/mediqa_qa` mirrors the GitHub XML. datasets-server counts 104 LiveQA-Med train questions, 104 Alexa train questions, 25 validation questions and 150 test questions. W19-5039 pairs those with 839, 862, 234 and 1,107 CHiQA answers. lm-eval scores `test` at question granularity. The authors' GitHub README states CC BY 4.0. The Hugging Face card lists the licence as unknown. GitHub's licence API field is null. Answers and ranks are public.\n\n## Who publishes it\n\nAsma Ben Abacha and Dina Demner-Fushman (Lister Hill Center, NLM) and Chaitanya Shivade (IBM) organised MEDIQA 2019 at ACL-BioNLP in Florence, August 2019. The overview is ACL Anthology W19-5039. EleutherAI ships the lm-eval task. The 2019 ranking board remains on AIcrowd.\n\n## Lineage\n\nDo not fold this into [MedQA](medqa.md). MedQA is USMLE-style multiple choice. One training file reuses TREC-2017 LiveQA medical questions, documented here as [live_qa](live_qa.md); lm-eval can load that split but scores `test`. HELM [medi_qa](medi_qa.md) is the jury wrap of the same test questions, not this overlap wrap. Later MEDIQA-Chat and MEDIQA-Sum tasks are different datasets.\n\n## Saturation and contamination\n\nSaturation of the lm-eval overlap metrics is not established. The 2019 ranking ceiling does not transfer. Contamination risk is high: the labelled test XML has been public since 2019. The perplexity YAML sets a decontamination query; the generation YAML does not.\n\n## How to run it\n\n```\nlm_eval --model hf --model_args pretrained=... --tasks mediqa_qa2019\n```\n\nThe companion task is `mediqa_qa2019_perplexity`. The loader uses default config `mediqa_qa_source`. Metrics need the `evaluate` extras named in `utils.py` (BLEU, ROUGE, BERTScore, BLEURT). Prompt wording, stop sequences and the `AnswerList[0]` gold make these numbers hard to compare with HELM `medi_qa` or with 2019 ranking accuracy.\n\n## Reading the numbers\n\nA high BLEU or BERTScore here means the generated English answer overlaps the first listed 2019 candidate, not that the model ranked CHiQA hits well. It is not a medical-licence exam score. Pair it with [medi_qa](medi_qa.md) if you need the jury protocol, and with a held-out clinical QA set if you need current medical knowledge. Do not quote the 78.3% ranking figure as this task.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mediqa_qa2019",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "mediqa_qa2019_perplexity",
   "MEDIQA-QA 2019 (lm-eval)"
  ],
  "category": "domain",
  "contamination": {
   "note": "Test questions, candidate answers and labels have been public on GitHub since 2019, including MEDIQA2019-Task3-QA-TestSet-wLabels.xml. lm-eval also sets should_decontaminate on the perplexity variant only.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC-BY-4.0 on the authors' GitHub README; Hugging Face bigbio/mediqa_qa lists licence as unknown",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 150,
   "size_note": "lm-eval scores the test split of bigbio/mediqa_qa (150 questions). Hugging Face datasets-server also reports train_live_qa_med 104, train_alexa 104, and validation 25 under both mediqa_qa_source and mediqa_qa_bigbio_qa. The YAML sets those splits but the generation gold is always the first AnswerList entry. Test XML is MEDIQA2019-Task3-QA-TestSet-wLabels.xml. W19-5039 reports 1,107 associated test answers (839 LiveQA-Med train answers, 862 Alexa train answers, 234 validation answers). lm-eval still scores one generated string per question.\n",
   "splits": "lm-eval: train_live_qa_med / validation / test (150-question test scored). Hugging Face also has train_alexa (104).",
   "url": "https://huggingface.co/datasets/bigbio/mediqa_qa"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-058 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-058"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "mediqa_qa2019",
   "opencompass": "",
   "other": "Also mediqa_qa2019_perplexity (loglikelihood_rolling). Dataset path bigbio/mediqa_qa, default Hugging Face config mediqa_qa_source. Distinct from HELM run spec medi_qa, which uses an LLM jury against the rank-1 reference.\n"
  },
  "id": "mediqa_qa2019",
  "last_updated": "",
  "leaderboard_url": "https://www.aicrowd.com/challenges/mediqa-2019-question-answering-qa/leaderboards",
  "lineage": {
   "family": "",
   "predecessor": "live_qa",
   "successors": [],
   "variants": [
    "medi_qa"
   ]
  },
  "measures": "This id is EleutherAI lm-evaluation-harness task mediqa_qa2019, not the 2019 ranking shared task as originally scored, and not HELM medi_qa. MEDIQA 2019 Task 3 gave a consumer health question plus CHiQA candidate answers and asked systems to filter and re-rank them. lm-eval instead takes the question text and asks the model to generate a free-form answer. Inputs and outputs are English text. The original shared task also had medical NLI and recognizing-question-entailment tracks; those are not this id.\n",
  "metric": {
   "baseline_note": "lm-eval logs six overlap metrics with nanmean aggregation. None is marked as the main score in the YAML. BLEURT uses bleurt-base-512; BERTScore is English F1. The 2019 shared task scored ranking with accuracy, MRR, precision and Spearman's rho and quoted 78.3% among 72 teams. Those ranking numbers are not on this overlap scale. No random or human baseline is stated for the generation protocol.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "bleu, rouge1, rouge2, rougeL, bleurt, bert_score (no designated headline)",
   "random_baseline": null,
   "unit": ""
  },
  "name": "MEDIQA 2019 QA (lm-eval)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "Overview of the MEDIQA 2019 Shared Task on Textual Inference, Question Entailment and Question Answering",
   "url": "https://aclanthology.org/W19-5039/",
   "year": 2019
  },
  "publisher": {
   "authors": [
    "Asma Ben Abacha",
    "Chaitanya Shivade",
    "Dina Demner-Fushman"
   ],
   "org": "U.S. National Library of Medicine (LHC/NLM) and IBM Research; lm-eval task by EleutherAI",
   "url": "https://sites.google.com/view/mediqa2019"
  },
  "released": "2019-08",
  "repo_url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/mediqa_qa2019",
  "saturation": {
   "as_of": "",
   "note": "The 2019 ranking QA track reached 78.3% among 72 teams. No current top was read for lm-eval BLEU/ROUGE/BLEURT/BERTScore on this generation wrap.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness mediqa_qa2019 task directory",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/mediqa_qa2019"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval mediqa_qa2019 README",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/mediqa_qa2019/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval mediqa_qa2019.yaml",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/mediqa_qa2019/mediqa_qa2019.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval mediqa_qa2019 utils.py (gold = AnswerList[0])",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/mediqa_qa2019/utils.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "bigbio/mediqa_qa dataset card",
    "url": "https://huggingface.co/datasets/bigbio/mediqa_qa"
   },
   {
    "accessed": "2026-09-08",
    "title": "bigbio/mediqa_qa datasets-server split counts",
    "url": "https://datasets-server.huggingface.co/info?dataset=bigbio/mediqa_qa"
   },
   {
    "accessed": "2026-09-08",
    "title": "bigbio/mediqa_qa loader (default config mediqa_qa_source)",
    "url": "https://huggingface.co/datasets/bigbio/mediqa_qa/raw/main/mediqa_qa.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "abachaa/MEDIQA2019 repository (CC BY 4.0 in README)",
    "url": "https://github.com/abachaa/MEDIQA2019"
   },
   {
    "accessed": "2026-09-08",
    "title": "MEDIQA2019 GitHub README (licence and 72-team note)",
    "url": "https://raw.githubusercontent.com/abachaa/MEDIQA2019/master/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "MEDIQA 2019 overview (ACL Anthology W19-5039)",
    "url": "https://aclanthology.org/W19-5039/"
   },
   {
    "accessed": "2026-09-08",
    "title": "MEDIQA 2019 overview PDF (150 test questions, 1107 answers; QA metrics)",
    "url": "https://aclanthology.org/W19-5039.pdf"
   },
   {
    "accessed": "2026-09-08",
    "title": "MEDIQA 2019 shared-task site",
    "url": "https://sites.google.com/view/mediqa2019"
   }
  ],
  "status": "unknown",
  "subcategory": "consumer health question answering (generation overlap metrics)",
  "summary": "lm-eval's generation wrap of MEDIQA 2019 Task 3: write an English answer to a consumer health question and score overlap against the first listed gold answer.",
  "tags": [
   "medical",
   "consumer-health",
   "generation",
   "overlap-metrics",
   "lm-eval"
  ],
  "task_format": "generate_until generation, stopping at a blank line. The YAML description tells the model to answer a patient question as a doctor would. Gold is QUESTION.AnswerList[0].Answer.AnswerText from the Hugging Face source schema, which is XML file order, not the expert's ReferenceRank=1 answer. A separate task mediqa_qa2019_perplexity scores loglikelihood of that gold string.\n"
 }
}