{
 "body": "\n## What it measures\n\nMedMCQA tests whether a model can answer multiple-choice medical questions drawn from two of India's\nlargest postgraduate medical entrance examinations, AIIMS and NEET PG. Each question covers one of 21\nmedical subjects (anatomy, pharmacology, surgery, obstetrics, and so on) and asks for the single best\nanswer among several options, mirroring the format used to screen doctors applying for postgraduate\nspecialty training in India. It is a single-turn, English-language, text-only task; the authors report\nit requires more than ten distinct types of reasoning across the question set, from single-fact recall\nto multi-hop clinical reasoning.\n\nBecause the exams it draws from are specific to the Indian medical curriculum, MedMCQA's subject mix and\nphrasing differ somewhat from the US-focused MedQA, even though both are \"medical multiple-choice\"\nbenchmarks.\n\n## How it is scored\n\nModels are scored on accuracy over four answer options, so random guessing scores 25%. The dataset ships\nwith a training split, a validation split, and a test split whose ground-truth labels are withheld by the\nauthors specifically to preserve the leaderboard's integrity; researchers submit predictions through a\nform rather than scoring locally against the test set. In practice, most automated evaluations, including\nEleutherAI's lm-evaluation-harness, score models against the public validation split instead, using it as\na stand-in test set, so a \"MedMCQA\" number in a model card is more often a validation-split score than a\ntrue held-out test score.\n\n## Dataset and licence\n\nMedMCQA contains 182,822 training, 6,150 validation and 4,183 test questions, a total of 193,155 (the\npaper's abstract rounds this to \"more than 194k\"), collected from AIIMS and NEET PG exam question banks\ncovering around 2,400 healthcare topics. The dataset and code are released under the MIT licence, and the\nauthors distribute both through a GitHub repository and a project site (medmcqa.github.io) that also\nhosts the leaderboard and submission form. Only training and validation examples ship with answers;\ntest-set answers are held by the authors.\n\n## Who publishes it\n\nMedMCQA was introduced by Ankit Pal, Logesh Kumar Umapathi and Malaikannan Sankarasubbu of Saama AI\nResearch in Chennai, India, and published at the 2022 ACM Conference on Health, Inference, and Learning\n(CHIL), with a preprint posted to arXiv in March 2022. The authors maintain the reference dataset, code\nand leaderboard themselves; no independent tracker for MedMCQA was found during this research.\n\n## Lineage\n\nMedMCQA has no formal predecessor or successor. It is one of the components Google bundled into its\n\"MultiMedQA\" evaluation suite for Med-PaLM and Med-PaLM 2, alongside MedQA and PubMedQA (both of which\nalso have pages in this repository), and it is regularly reported together with those two by\nmedical-specialist model developers such as Google's MedGemma team. It is not a variant of MedQA: the two\nwere built independently, from different national exam systems, by different research groups.\n\n## Saturation and contamination\n\nThe project's own leaderboard, checked during this research, still shows a top dev-accuracy figure\naround 63%, with no date attached, reflecting BERT-style and early GPT-era submissions rather than\ncurrent frontier models. More recent individual model reports place scores well above that: Google's\nMedGemma Technical Report put its 27B model at 74.2% in July 2025. Contamination risk sits at medium:\nthe authors' withheld test set carries low risk if it is actually used, but the training and validation\nsplits, including the validation split most automated harnesses score against, have been fully public\nwith answers since March 2022, more than four years before this research.\n\n## How to run it\n\nEleutherAI's lm-evaluation-harness implements the task as `medmcqa`, reading the\nopenlifescienceai/medmcqa mirror on Hugging Face and, notably, setting both its validation and test\nsplit fields to the dataset's public validation split, since the authors do not distribute test-set\nlabels. It scores accuracy and normalized accuracy. Because the harness effectively evaluates against a\ndifferent split than the one the authors call \"test,\" and because the authors' own leaderboard requires\nan ungated prediction submission rather than local scoring, scores reported as \"MedMCQA\" can come from\neither protocol and are not automatically comparable.\n\n## Reading the numbers\n\nA high MedMCQA score means a model reliably selects the answer intended by Indian postgraduate medical\nentrance exam writers, not that it can practise medicine safely. Developers who report it alongside\nMedQA and PubMedQA, such as Google's MedGemma team, pair it with an explicit disclaimer that outputs\n\"are not intended to directly inform clinical diagnosis, patient management decisions, treatment\nrecommendations, or any other direct clinical practice applications\" and require independent clinical\nverification. Because most published scores are actually validation-split scores rather than scores\nagainst the authors' withheld test set, treat small differences between models with some caution, and\nread MedMCQA alongside MedQA and PubMedQA rather than alone.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "medmcqa",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "medgemma 27B it",
   "model_id": "google/medgemma-27b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 74.2,
   "source": "domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 27B",
   "model_id": "google/gemma-3-27b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 62.6,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2,, domain-evals llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "medgemma 1.5 4B it",
   "model_id": "google/medgemma-1-5-4b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 55.7,
   "source": "domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "medgemma 4B it",
   "model_id": "google/medgemma-4b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 55.7,
   "source": "domain-evals"
  }
 ],
 "page": {
  "aliases": [
   "Med-MCQA"
  ],
  "category": "domain",
  "contamination": {
   "note": "The authors' withheld test set carries low risk if it is actually used, since its answers are not public. But the training and validation splits, including the validation split most automated harnesses score against as a proxy test set, have been fully public with answers since March 2022, more than four years before this research, long enough for likely inclusion in large training corpora.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": 193155,
   "size_note": "182,822 training, 6,150 validation and 4,183 test questions, summing to 193,155; the paper's own abstract rounds this to \"more than 194k.\" Only training and validation examples ship with answers; the authors withhold test-set ground truth specifically to preserve leaderboard integrity, so most automated evaluations (including EleutherAI's lm-evaluation-harness) score against the public validation split as a proxy test set instead.\n",
   "splits": "182,822 train / 6,150 validation / 4,183 test (test-set answers withheld by the authors)",
   "url": "https://github.com/MedMCQA/MedMCQA"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 1b, slice O",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "medmcqa",
   "opencompass": "",
   "other": "lm-evaluation-harness's medmcqa task reads the openlifescienceai/medmcqa mirror on Hugging Face and sets both its \"validation\" and \"test\" split fields to the dataset's public validation split, since the authors do not distribute test-set labels; it scores accuracy and normalized accuracy.\n"
  },
  "id": "medmcqa",
  "last_updated": "",
  "leaderboard_url": "https://medmcqa.github.io/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "MedMCQA tests whether a model can answer multiple-choice medical questions drawn from two of India's largest postgraduate medical entrance examinations, AIIMS and NEET PG. Each question covers one of 21 medical subjects (anatomy, pharmacology, surgery, obstetrics, and so on) and asks for the single best answer among several options, mirroring the format used to screen doctors applying for postgraduate specialty training in India. It is a single-turn, English-language, text-only task; the authors report it requires more than ten distinct types of reasoning across the question set, from single-fact recall to multi-hop clinical reasoning. Because the exams it draws from are specific to the Indian medical curriculum, its subject mix and phrasing differ somewhat from the US-focused MedQA, even though both are \"medical multiple-choice\" benchmarks.\n",
  "metric": {
   "baseline_note": "No physician or expert human baseline was established from the sources reviewed for this page.\n",
   "direction": "higher_is_better",
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 25,
   "unit": "%"
  },
  "name": "MedMCQA",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2203.14371",
   "title": "MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering",
   "url": "https://arxiv.org/abs/2203.14371",
   "year": 2022
  },
  "publisher": {
   "authors": [
    "Ankit Pal",
    "Logesh Kumar Umapathi",
    "Malaikannan Sankarasubbu"
   ],
   "org": "Saama AI Research",
   "url": "https://github.com/MedMCQA/MedMCQA"
  },
  "released": "2022-03",
  "repo_url": "https://github.com/MedMCQA/MedMCQA",
  "saturation": {
   "as_of": "2025-07",
   "note": "The project's own leaderboard, checked during this research, still shows a top dev-accuracy figure around 63%, with no date attached, reflecting BERT-style and early GPT-era submissions rather than current frontier models. Google's MedGemma Technical Report reported a materially higher 74.2% for its 27B model in July 2025, suggesting the official leaderboard has not kept pace with current model reports and should not be read as a live ranking of frontier models.\n",
   "status": "watch",
   "top_score": 74.2
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering",
    "url": "https://arxiv.org/abs/2203.14371"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedMCQA/MedMCQA GitHub repository",
    "url": "https://github.com/MedMCQA/MedMCQA"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedMCQA project site and leaderboard",
    "url": "https://medmcqa.github.io/"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering, Proceedings of CHIL 2022",
    "url": "https://proceedings.mlr.press/v174/pal22a.html"
   },
   {
    "accessed": "2026-09-08",
    "title": "medmcqa task config, EleutherAI lm-evaluation-harness",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/medmcqa/medmcqa.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedGemma Technical Report",
    "url": "https://arxiv.org/abs/2507.05201"
   }
  ],
  "status": "active",
  "subcategory": "medical entrance exam question answering",
  "summary": "Over 194,000 multiple-choice questions from India's AIIMS and NEET PG medical entrance exams, across 21 subjects.",
  "tags": [
   "medical",
   "multiple-choice",
   "medical-entrance-exam",
   "india"
  ],
  "task_format": "Four-option multiple-choice medical exam question; the model returns a single letter answer (A-D).\n"
 }
}