{
 "body": "\n## What it measures\n\nPubMedQA tests whether a model can answer a yes/no/maybe research question using the PubMed abstract\nthat the question was derived from. Each item pairs a question phrased from a paper's own title or\nconclusion (for example, \"Do preoperative statins reduce atrial fibrillation after coronary artery\nbypass grafting?\") with that paper's abstract, and asks the model to decide whether the abstract's\nevidence supports a yes, no, or maybe/inconclusive answer. It is a single-turn, English-language,\ntext-only reading-comprehension task grounded in biomedical research literature rather than clinical\npractice or exam questions, which sets it apart from MedQA and MedMCQA.\n\n## How it is scored\n\nModels are graded on three-way classification accuracy (yes/no/maybe). A model guessing uniformly at\nrandom across the three labels would score roughly 33%, but the classes are not evenly distributed: the\noriginal paper's own simple baseline, driven by the imbalanced \"yes\" majority class, already scores\n55.2%, against a best-model score of 68.1% and a human performance ceiling of 78.0% reported in the\npaper. Most current evaluations use only the 1,000-item expert-labelled subset (PQA-L) as the scored\nset, commonly following a ten-fold cross-validation protocol; EleutherAI's lm-evaluation-harness, for\ninstance, evaluates against a single fixed fold (fold 0) of that split rather than all ten.\n\n## Dataset and licence\n\nThe full PubMedQA release totals 273,518 question-answer instances across three subsets: 1,000\nexpert-annotated instances (PQA-L, the subset used for reported accuracy scores), 61,200 unlabelled\ninstances (PQA-U) and 211,300 artificially generated instances (PQA-A, labelled automatically from paper\nconclusions rather than by human annotators). Only PQA-L functions as an evaluation set; PQA-U and PQA-A\nare intended as additional pretraining or weak-supervision data. The dataset and code are released under\nthe MIT licence and hosted on GitHub and Hugging Face, with answers included in the public PQA-L files.\n\n## Who publishes it\n\nPubMedQA was introduced by Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen and Xinghua Lu,\naffiliated with the University of Pittsburgh, Carnegie Mellon University and Google AI, and presented at\nEMNLP-IJCNLP 2019. The dataset and an accuracy leaderboard are maintained by the authors at\npubmedqa.github.io, where submissions are still accepted by email; the site records its leaderboard as\nlast updated in April 2024.\n\n## Lineage\n\nPubMedQA has no formal predecessor or successor of its own. Like MedQA and MedMCQA, both of which have\npages in this repository, it is one of the components Google folded into its \"MultiMedQA\" evaluation\nsuite for Med-PaLM and Med-PaLM 2, and the three are commonly reported together by medical-specialist\nmodel developers. It is not a variant of either: all three were built independently, by different\ngroups, from different source material (licensing exams versus research abstracts).\n\n## Saturation and contamination\n\nPubMedQA's own leaderboard shows GPT-4 with Microsoft's Medprompt technique leading at 82.0% accuracy in\nthe \"reasoning-required\" setting as of November 2023, already above the 78.0% human performance figure\nthe original 2019 paper reported \u2014 a rare case among the benchmarks on this site where a widely cited\nscore has clearly passed the paper's own human baseline. Not every later model has kept pace, though:\nGoogle's MedGemma Technical Report reported 76.8% (text-only) and 77.2% (multimodal) for its 27B model in\nJuly 2025, both below the 2023 GPT-4-Medprompt score. Contamination risk is high: the labelled evaluation\nset has been public with answers since September 2019, and the larger artificially labelled subset is\ncommonly used as biomedical fine-tuning data in its own right, both circumstances that make clean\nheld-out evaluation increasingly unlikely seven years on.\n\n## How to run it\n\nEleutherAI's lm-evaluation-harness implements the task as `pubmedqa`, reading the bigbio/pubmed_qa\nmirror's `pubmed_qa_labeled_fold0_source` configuration and scoring three-way (yes/no/maybe) accuracy\nagainst that single cross-validation fold rather than the authors' own ten-fold protocol. Because the\nauthors' leaderboard, the ten-fold protocol, and the harness's single-fold shortcut are three different\nevaluation setups, and because \"reasoning-required\" and \"reasoning-free\" variants exist depending on\nwhether the model also sees the paper's long-form answer, PubMedQA scores from different sources need\ntheir protocol checked before they are compared.\n\n## Reading the numbers\n\nA high PubMedQA score shows a model can extract a yes/no/maybe verdict from a research abstract the way\nthe paper's own conclusion does, not that it can appraise biomedical evidence the way a researcher or\nclinician would, and certainly not that it is fit for clinical decision-making. Google, which reports\nPubMedQA for MedGemma alongside MedQA and MedMCQA, states plainly that model outputs \"are not intended to\ndirectly inform clinical diagnosis, patient management decisions, treatment recommendations, or any\nother direct clinical practice applications\" and require independent clinical verification. Because the\nbenchmark's top scores already sit at or above its own reported human baseline, treat further gains as\ndiminishing in significance, and check whether a reported number came from the reasoning-required or\nreasoning-free protocol before comparing it to another model's.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "pubmedqa",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "medgemma 27B it",
   "model_id": "google/medgemma-27b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 76.8,
   "source": "domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 27B",
   "model_id": "google/gemma-3-27b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 73.4,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2,, domain-evals llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "medgemma 1.5 4B it",
   "model_id": "google/medgemma-1-5-4b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 73.4,
   "source": "domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "medgemma 4B it",
   "model_id": "google/medgemma-4b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 73.4,
   "source": "domain-evals"
  }
 ],
 "page": {
  "aliases": [
   "PQA-L"
  ],
  "category": "domain",
  "contamination": {
   "note": "The 1,000-item labelled evaluation set, with answers included, has been public since September 2019. The larger 211,300-item artificially labelled subset is also commonly used as biomedical fine-tuning data in its own right. Neither carries a canary string or gating mechanism, which makes clean held-out evaluation increasingly unlikely this many years after release.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 1000,
   "size_note": "The 1,000-item expert-annotated subset (PQA-L) is the evaluation set used for reported accuracy scores. The full release totals 273,518 instances: PQA-L (1,000, expert-labelled) plus two non-evaluation subsets intended for training/weak supervision, PQA-U (61,200 unlabelled) and PQA-A (211,300 artificially generated, labelled automatically from paper conclusions).\n",
   "splits": "1,000 expert-labelled questions (PQA-L) used for evaluation, commonly reported via ten-fold cross-validation; EleutherAI's lm-evaluation-harness instead evaluates a single fixed fold (fold 0). Exact per-fold train/validation/test counts were not established from the sources reviewed for this page.\n",
   "url": "https://huggingface.co/datasets/qiaojin/PubMedQA"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 1b, slice O",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "pubmedqa",
   "opencompass": "",
   "other": "lm-evaluation-harness's pubmedqa task reads the bigbio/pubmed_qa mirror's pubmed_qa_labeled_fold0_source configuration and scores three-way (yes/no/maybe) accuracy against that single cross-validation fold rather than the authors' own ten-fold protocol.\n"
  },
  "id": "pubmedqa",
  "last_updated": "",
  "leaderboard_url": "https://pubmedqa.github.io",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "PubMedQA tests whether a model can answer a yes/no/maybe research question using the PubMed abstract that the question was derived from. Each item pairs a question phrased from a paper's own title or conclusion (for example, \"Do preoperative statins reduce atrial fibrillation after coronary artery bypass grafting?\") with that paper's abstract, and asks the model to decide whether the abstract's evidence supports a yes, no, or maybe/inconclusive answer. It is a single-turn, English-language, text-only reading-comprehension task grounded in biomedical research literature rather than clinical practice or exam questions, which sets it apart from MedQA and MedMCQA.\n",
  "metric": {
   "baseline_note": "The original paper reports a human performance ceiling of 78.0% and its own simple baseline model (driven by the imbalanced \"yes\" majority class rather than blind guessing) at 55.2%, both higher than the 33.3% chance rate for uniform random guessing across three labels.\n",
   "direction": "higher_is_better",
   "human_baseline": 78.0,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 33.3,
   "unit": "%"
  },
  "name": "PubMedQA",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "1909.06146",
   "title": "PubMedQA: A Dataset for Biomedical Research Question Answering",
   "url": "https://arxiv.org/abs/1909.06146",
   "year": 2019
  },
  "publisher": {
   "authors": [
    "Qiao Jin",
    "Bhuwan Dhingra",
    "Zhengping Liu",
    "William W. Cohen",
    "Xinghua Lu"
   ],
   "org": "University of Pittsburgh",
   "url": "https://pubmedqa.github.io"
  },
  "released": "2019-09",
  "repo_url": "https://github.com/pubmedqa/pubmedqa",
  "saturation": {
   "as_of": "2023-11",
   "note": "PubMedQA's own leaderboard shows GPT-4 with Microsoft's Medprompt technique leading at 82.0% accuracy in the \"reasoning-required\" setting as of November 2023, already above the 78.0% human performance figure the original 2019 paper reported. Not every later model has kept pace: Google's MedGemma Technical Report reported 76.8% (text-only) and 77.2% (multimodal) for its 27B model in July 2025, both below the 2023 GPT-4-Medprompt figure and the human baseline.\n",
   "status": "saturated",
   "top_score": 82.0
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "PubMedQA: A Dataset for Biomedical Research Question Answering",
    "url": "https://arxiv.org/abs/1909.06146"
   },
   {
    "accessed": "2026-09-08",
    "title": "pubmedqa/pubmedqa GitHub repository",
    "url": "https://github.com/pubmedqa/pubmedqa"
   },
   {
    "accessed": "2026-09-08",
    "title": "PubMedQA project site and leaderboard",
    "url": "https://pubmedqa.github.io"
   },
   {
    "accessed": "2026-09-08",
    "title": "qiaojin/PubMedQA dataset card, Hugging Face",
    "url": "https://huggingface.co/datasets/qiaojin/PubMedQA"
   },
   {
    "accessed": "2026-09-08",
    "title": "pubmedqa task config, EleutherAI lm-evaluation-harness",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/pubmedqa/pubmedqa.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "PubMedQA: A Dataset for Biomedical Research Question Answering, ACL Anthology (EMNLP-IJCNLP 2019)",
    "url": "https://aclanthology.org/D19-1259/"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedGemma Technical Report",
    "url": "https://arxiv.org/abs/2507.05201"
   }
  ],
  "status": "active",
  "subcategory": "biomedical literature question answering",
  "summary": "Yes/no/maybe research questions answered from their source PubMed abstract, testing biomedical reading comprehension.",
  "tags": [
   "medical",
   "biomedical-literature",
   "yes-no-maybe",
   "reading-comprehension"
  ],
  "task_format": "Three-way yes/no/maybe classification given a research question and its source PubMed abstract.\n"
 }
}