{
 "body": "\n## What it measures\n\nMedicationQA asks a model to answer a real consumer question about a drug. The questions were\nsubmitted to MedlinePlus, then filtered so that a drug name is the focus and the wording is\nanswerable. Types span information, dose, usage, side effects, indication, interaction and other\nlabels, 25 in all. Each gold item has one reference answer that annotators retrieved from\nDailyMed, MedlinePlus, other NIH or government sites, or a short list of trusted publishers. The\ntask is English, open-ended, and about medication facts a patient might ask, not a clinical\nvignette or an exam item.\n\nHELM uses that gold file as a generation benchmark. The model sees only the question. It does not\nsee the candidate pages the original annotators searched.\n\n## How it is scored\n\nHELM's main metric is `medication_qa_accuracy`, an LLM-jury mean of accuracy, completeness and\nclarity, each on a 1-5 scale against the gold answer. ROUGE and BERTScore are logged as well. The\n2019 paper does not publish an LLM answering score. It reports a Bi-LSTM-CRF for drug-name focus\nrecognition and a CNN for question type, plus a qualitative look at CHiQA. Those figures are not\nsubstitutes for the HELM jury score. No random-guess or human answering baseline was found.\n\n## Dataset and licence\n\nThe paper and README state 674 pairs. DailyMed supplied 290 answers, MedlinePlus 128, and other\nsites 256. There is no official train/dev/test split. The xlsx used range is A1:F691, and every\none of the 690 data rows had an Answer cell in this review, so a HELM `Answer.isna()` filter would\nkeep 690 items rather than 674. The GitHub README licences the dataset as CC BY 4.0. The MEDINFO\n2019 article PDF is marked CC BY-NC 4.0. Use the dataset licence for the spreadsheet and do not\ntreat the article licence as a second dataset term.\n\n## Who publishes it\n\nAsma Ben Abacha, Yassine Mrabet, Mark Sharp, Travis Goodwin, Sonya E. Shooshan and Dina\nDemner-Fushman at NLM's Lister Hill Center released the corpus with the MEDINFO 2019 paper\n(DOI 10.3233/SHTI190176). Stanford CRFM added the HELM scenario. No standing public answering\nleaderboard was found.\n\n## Lineage\n\nThis is not [MedQA](medqa.md) and not [medi_qa](medi_qa.md). MedQA is USMLE multiple choice.\n`medi_qa` is HELM's wrap of MEDIQA 2019 Task 3, a ranking set built from CHiQA output. MedicationQA\nis a later NLM gold set of medication questions only. [MeQSum](meqsum.md) summarises long consumer\nquestions; it does not answer them. [live_qa](live_qa.md) is the TREC-2017 LiveQA medical test set.\n\n## Saturation and contamination\n\nNo current HELM leaderboard number was established here, so saturation is unknown. Contamination\nrisk is high: questions, answers and URLs have been public since 2019.\n\n## How to run it\n\nIn HELM, the run spec is `medication_qa`. It generates zero-shot and annotates with\n`MedicationQAAnnotator`. Judge models come from HELM's jury config; the populated default was not\nopened here. Compare only jury scores with other HELM MedHELM jury scores, not with 2019 CNN type\naccuracy.\n\n## Reading the numbers\n\nA high HELM score means judge models thought the generated answer matched a trusted drug-label\nsnippet on accuracy, completeness and clarity. It does not mean the model is safe to advise\npatients, and it does not test exam-style diagnosis. Many gold answers are copied from DailyMed or\nMedlinePlus, so overlap with those pages can inflate a score. Read it next to MEDIQA and LiveQA\nfor broader consumer QA, not as a clinical-reasoning exam.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "medication_qa",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Medication_QA",
   "Medication QA MedInfo 2019",
   "MedInfo2019-QA-Medications"
  ],
  "category": "domain",
  "contamination": {
   "note": "The gold spreadsheet has been public on GitHub since 2019, with questions, answers and source URLs ungated.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC-BY-4.0 on the GitHub README for the dataset; the MEDINFO 2019 article itself is CC BY-NC 4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 674,
   "size_note": "The paper and GitHub README state 674 question-answer pairs. Answers were taken from DailyMed (290), MedlinePlus (128) and other sites (256). HELM downloads MedInfo2019-QA-Medications.xlsx and drops rows whose Answer cell is pandas-NA, then scores every remaining row as test. The spreadsheet used range is A1:F691 (header plus 690 data rows). In this review every data row had a non-empty Answer cell, so HELM would keep 690 rows, 16 more than the published 674. Mean token lengths from the paper's Table 2 were not recovered from the PDF text layer.\n",
   "splits": "No official train/dev/test split; HELM treats the filtered spreadsheet as a single test set.",
   "url": "https://github.com/abachaa/Medication_QA_MedInfo2019"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch 7 pilot (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, pilot-review"
  },
  "harness": {
   "bigbench": "",
   "helm": "medication_qa",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "HELM run spec `medication_qa` loads MedicationQAScenario, generates zero-shot, annotates with MedicationQAAnnotator, and reports medication_qa_accuracy plus summarisation metrics.\n"
  },
  "id": "medication_qa",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "MedicationQA tests whether a model can answer a real consumer question about a drug in plain English. Questions come from MedlinePlus users and always have a drug as the focus. Types include information, dose, usage, side effects, indication and interaction, among 25 labels in the gold standard. Each item pairs the question with one expert-retrieved reference answer and its source URL. HELM treats the whole set as a zero-shot generation task. It is not MedQA, not MEDIQA 2019 ranking, and not a multiple-choice exam.\n",
  "metric": {
   "baseline_note": "HELM's main metric is an LLM jury on 1-5 ratings of accuracy, completeness and clarity, plus ROUGE and BERTScore. The 2019 paper does not report end-to-end answer accuracy for modern LLMs; it reports Bi-LSTM-CRF focus recognition and CNN question-type identification on this corpus. No random or human answering baseline was established from the sources reviewed.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 5,
   "name": "medication_qa_accuracy (HELM LLM-jury average of accuracy, completeness and clarity, each 1-5)",
   "random_baseline": null,
   "unit": "points"
  },
  "name": "MedicationQA",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "Bridging the Gap Between Consumers' Medication Questions and Trusted Answers",
   "url": "https://pubmed.ncbi.nlm.nih.gov/31437878/",
   "year": 2019
  },
  "publisher": {
   "authors": [
    "Asma Ben Abacha",
    "Yassine Mrabet",
    "Mark Sharp",
    "Travis Goodwin",
    "Sonya E. Shooshan",
    "Dina Demner-Fushman"
   ],
   "org": "Lister Hill National Center for Biomedical Communications, U.S. National Library of Medicine; HELM scenario by Stanford CRFM",
   "url": "https://github.com/abachaa/Medication_QA_MedInfo2019"
  },
  "released": "2019",
  "repo_url": "https://github.com/abachaa/Medication_QA_MedInfo2019",
  "saturation": {
   "as_of": "",
   "note": "No standing LLM leaderboard for HELM's medication_qa_accuracy was found during this research. The 2019 paper's CNN question-type accuracy (75.7% on a 14-type collapse) is not an answering score and is not recorded as top_score.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "abachaa/Medication_QA_MedInfo2019 GitHub repository",
    "url": "https://github.com/abachaa/Medication_QA_MedInfo2019"
   },
   {
    "accessed": "2026-09-08",
    "title": "PubMed record for the MEDINFO 2019 MedicationQA paper",
    "url": "https://pubmed.ncbi.nlm.nih.gov/31437878/"
   },
   {
    "accessed": "2026-09-08",
    "title": "NLM PDF of Bridging the Gap Between Consumers' Medication Questions and Trusted Answers",
    "url": "https://lhncbc.nlm.nih.gov/LHC-publications/PDF/pub9965.pdf"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM MedicationQAScenario source",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/medication_qa_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM medhelm_run_specs.py (medication_qa run spec)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/run_specs/medhelm_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM MedicationQAAnnotator source",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/annotation/medication_qa_annotator.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "IOS Press DOI 10.3233/SHTI190176",
    "url": "https://doi.org/10.3233/SHTI190176"
   }
  ],
  "status": "active",
  "subcategory": "consumer medication question answering",
  "summary": "Open-ended consumer questions about medications paired with trusted reference answers drawn from DailyMed, MedlinePlus and similar sources.",
  "tags": [
   "medical",
   "consumer-health",
   "medication",
   "generation",
   "llm-judge",
   "helm"
  ],
  "task_format": "Zero-shot generation. HELM's instruction is \"Please answer the following consumer health question.\" The model writes free text (max 512 tokens in the HELM run spec).\n"
 }
}