{
 "body": "\n## What it measures\n\nMedQA tests whether a model can pick the correct answer to a clinical multiple-choice question written\nin the style of the United States Medical Licensing Examination. Most questions present a short patient\nvignette (age, presenting symptoms, exam findings, sometimes lab values) and ask for a diagnosis, a next\nstep in management, or an underlying mechanism, then offer several candidate answers. It is a\nsingle-turn, English-language, text-only task built from real practice-exam question banks rather than\nwritten for the benchmark, so it leans on applied clinical reasoning more than isolated fact recall.\n\nThe id on this page covers only the English/USMLE-style subset. The original 2020 release also published\nmuch larger Simplified Chinese (34,251 questions, from Mainland Chinese licensing exams) and Traditional\nChinese (14,123 questions, from Taiwanese exams) subsets; those are rarely reported in model cards and\nare not covered by this id.\n\n## How it is scored\n\nModels are graded on accuracy: the share of questions answered with the correct letter. Most reported\nscores use the four-option variant, where random guessing scores 25%. The source questions originally\nshipped with up to five options; the widely used four-option version was produced by randomly deleting\none incorrect option per question, so \"MedQA\" and \"MedQA 4-options\" scores are not always directly\ncomparable to older five-option numbers. Most evaluations run the model zero-shot or few-shot and\ncompare its selected letter against the labelled answer; some system cards instead report a best-of-N\nor self-consistency variant that samples several completions and takes a majority vote, which raises\nscores relative to a single greedy pass.\n\n## Dataset and licence\n\nThe evaluation subset most models report is the English-language question bank: 12,723 questions, split\n10,178/1,272/1,273 across train, validation and test on the commonly used four-option Hugging Face\nmirror. Questions were collected from professional medical board exam question banks rather than written\nfor the benchmark. The GitHub repository carries an MIT licence badge, but the paper states the\naccompanying medical-textbook corpus (included to support retrieval-based baselines) is released only\nunder a research-use-only agreement, so the textbooks' licensing differs from that on the question\npairs. Hugging Face's listing for the multilingual mirror records the licence as unspecified. Test-split\nanswers are included in the public files; there is no gating.\n\n## Who publishes it\n\nMedQA was introduced by Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang and Peter\nSzolovits, affiliated with MIT's Computer Science and Artificial Intelligence Laboratory, the MIT-IBM\nWatson AI Lab, and Dialectic Inc., and posted to arXiv in September 2020. The authors maintain the\nreference dataset and baseline code at github.com/jind11/MedQA. No single organisation runs an actively\nupdated public leaderboard for it today; current scores mostly surface inside individual model papers\nand system cards, including Google's Med-PaLM, Med-PaLM 2 and MedGemma reports.\n\n## Lineage\n\nMedQA has no formal predecessor or successor of its own, but Google bundled it into the \"MultiMedQA\"\nevaluation suite for Med-PaLM and Med-PaLM 2, alongside MedMCQA and PubMedQA (both of which also have\npages in this repository). It sits in the same broad space as MedMCQA (Indian medical entrance exams)\nand the newer, harder MedXpertQA (2025). None of these is a formal variant of MedQA: each was built\nindependently, with its own question sources and methodology.\n\n## Saturation and contamination\n\nReported scores have risen substantially since 2020. Google's Med-PaLM 2 reported 86.5% in May 2023;\nGoogle's MedGemma Technical Report reported 87.7% zero-shot and 89.8% with best-of-5 test-time scaling\nfor its 27B model in July 2025. That is a narrowing gap to a perfect score in just over two years, though\nno confirmed physician or passing-score baseline exists in the sources reviewed here to weigh it against.\nContamination risk is high: the English question-and-answer set, including test-split answers, has been\ndownloadable from GitHub without any gating or canary string since September 2020, giving it roughly six\nyears of exposure to web crawls and model training corpora.\n\n## How to run it\n\nEleutherAI's lm-evaluation-harness implements the standard four-option task as `medqa_4options`, reading\nthe GBaker/MedQA-USMLE-4-options-hf mirror and scoring accuracy and normalized accuracy over the\n1,273-question test split. Because the four-option filtering step was done independently of the paper's\noriginal authors, and because system cards vary in whether they report zero-shot, few-shot or best-of-N\nsampling, MedQA scores from different sources are not always computed the same way.\n\n## Reading the numbers\n\nA high MedQA score shows a model is good at picking the textbook-favoured answer to an exam-style\nclinical vignette under exam conditions, not that it is safe or competent to use in real patient care.\nGoogle draws exactly that line for MedGemma, which reports MedQA among its headline numbers: its model\ncard states that outputs \"are not intended to directly inform clinical diagnosis, patient management\ndecisions, treatment recommendations, or any other direct clinical practice applications,\" and that all\noutputs require independent clinical verification. Read a MedQA number alongside MedMCQA and PubMedQA,\nwhich test differently structured medical knowledge, and treat scores from different harnesses, shot\ncounts or sampling strategies as only roughly comparable.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "medqa",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o1",
   "model_id": "openai/o1",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 96.5,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 3.1 Pro Preview",
   "model_id": "google/gemini-3-1-pro-preview",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 96.4,
   "source": "anthropic-system-card-mythos, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.1",
   "model_id": "openai/gpt-5-1",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 96.4,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o3",
   "model_id": "openai/o3",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 96.1,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.2",
   "model_id": "openai/gpt-5-2",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.8,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o4-mini",
   "model_id": "openai/o4-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.2,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5",
   "model_id": "openai/gpt-5",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 93.0,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.4",
   "model_id": "openai/gpt-5-4",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 93.0,
   "source": "lmarena.ai, provider-reports, anthropic-system-card-mythos, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1",
   "model_id": "deepseek/deepseek-r1",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 92.1,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 0528",
   "model_id": "deepseek/deepseek-r1-0528",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 92.1,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o3-mini",
   "model_id": "openai/o3-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 91.4,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1",
   "model_id": "openai/gpt-4-1",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 89.7,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, domain-evals preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "medgemma 27B it",
   "model_id": "google/medgemma-27b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 88.5,
   "source": "domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.7",
   "model_id": "anthropic/claude-3-7-sonnet-20250219",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 87.6,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3",
   "model_id": "xai/grok-3",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 86.1,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 Max",
   "model_id": "qwen/qwen3-max",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 85.5,
   "source": "domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 30B-A3B",
   "model_id": "qwen/qwen3-30b-a3b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 85.3,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 235B-A22B",
   "model_id": "qwen/qwen3-235b-a22b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 84.8,
   "source": "lmarena.ai, provider-reports, preference-evals, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.0 Flash",
   "model_id": "google/gemini-2-0-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 83.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 405B Instruct",
   "model_id": "meta/llama-3-1-405b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 82.9,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4",
   "model_id": "anthropic/claude-opus-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 82.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.6",
   "model_id": "anthropic/claude-opus-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 82.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, anthropic-system-card-mythos"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large 2.1",
   "model_id": "mistral/mistral-large-2411",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 81.5,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals,, domain-evals open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3",
   "model_id": "deepseek/deepseek-v3",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 80.3,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, domain-evals open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro",
   "model_id": "google/gemini-2-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 80.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 05-06",
   "model_id": "google/gemini-2-5-pro-preview-05-06",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 80.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 06-05",
   "model_id": "google/gemini-2-5-pro-preview-06-05",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 80.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview TTS",
   "model_id": "google/gemini-2-5-pro-preview-tts",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 80.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Medium (latest)",
   "model_id": "mistral/mistral-medium-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 79.1,
   "source": "lmarena.ai, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Medium 3",
   "model_id": "mistral/mistral-medium-2505",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 79.1,
   "source": "lmarena.ai, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Medium 3.1",
   "model_id": "mistral/mistral-medium-2508",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 79.1,
   "source": "lmarena.ai, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o",
   "model_id": "openai/gpt-4o",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 78.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-05-13)",
   "model_id": "openai/gpt-4o-2024-05-13",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 78.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-08-06)",
   "model_id": "openai/gpt-4o-2024-08-06",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 78.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-11-20)",
   "model_id": "openai/gpt-4o-2024-11-20",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 78.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o mini",
   "model_id": "openai/gpt-4o-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 78.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Maverick 17B 128E Instruct",
   "model_id": "meta/llama-4-maverick-17b-128e-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 78.4,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Pixtral Large (latest)",
   "model_id": "mistral/pixtral-large-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 78.3,
   "source": "multimodal-evals, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3.5",
   "model_id": "anthropic/claude-3-5-haiku-20241022",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 77.8,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3.5 (latest)",
   "model_id": "anthropic/claude-3-5-haiku-latest",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 77.8,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 4.5",
   "model_id": "anthropic/claude-haiku-4-5-20251001",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 77.8,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 4.5 (latest)",
   "model_id": "anthropic/claude-haiku-4-5",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 77.8,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "phi 4",
   "model_id": "microsoft/phi-4",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 77.8,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2,, domain-evals llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 27B",
   "model_id": "google/gemma-3-27b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 74.9,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2,, domain-evals llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Command A Vision",
   "model_id": "cohere/command-a-vision-07-2025",
   "provider": "cohere",
   "provider_display": "Cohere",
   "score": 73.3,
   "source": "multimodal-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "medgemma 4B it",
   "model_id": "google/medgemma-4b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 72.1,
   "source": "domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 70B",
   "model_id": "meta/llama-3-1-70b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 65.2,
   "source": "lmarena.ai, provider-reports, safety-evals, domain-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 70B Instruct",
   "model_id": "meta/llama-3-1-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 65.2,
   "source": "lmarena.ai, provider-reports, safety-evals, domain-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "medgemma 1.5 4B it",
   "model_id": "google/medgemma-1-5-4b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 64.4,
   "source": "domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 3B Instruct",
   "model_id": "meta/llama-3-2-3b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 52.6,
   "source": "llm-stats, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Scout 17B 16E Instruct",
   "model_id": "meta/llama-4-scout-17b-16e-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 52.0,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, multimodal-evals, safety-evals"
  }
 ],
 "page": {
  "aliases": [
   "MedQA-USMLE",
   "MedQA-USMLE-4-options"
  ],
  "category": "domain",
  "contamination": {
   "note": "The English question-and-answer set, including test-split answers, has been publicly downloadable from GitHub without gating since September 2020, giving it about six years of exposure to web crawls and model training corpora as of this research. No canary string or access agreement covers the question set itself, only the separately licensed textbook corpus.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Not stated by the paper; the authors' GitHub repository is MIT-licensed; the accompanying textbook corpus is released under a research-use-only agreement",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 12723,
   "size_note": "12,723 English-language USMLE-style questions, matching both the original paper's reported count and the train/validation/test split sizes on the commonly used four-option Hugging Face mirror (10,178 / 1,272 / 1,273). The original release also contains 34,251 Simplified Chinese questions (from Mainland Chinese licensing exams) and 14,123 Traditional Chinese questions (from Taiwanese exams); those are separate subsets not covered by this page's id. Source questions were originally released with up to five options; the four-option variant used by lm-evaluation-harness and most model reports was produced by randomly deleting one wrong option per question.\n",
   "splits": "10,178 train / 1,272 validation / 1,273 test (four-option English mirror)",
   "url": "https://huggingface.co/datasets/GBaker/MedQA-USMLE-4-options-hf"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 1b, slice O",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "medqa_4options",
   "opencompass": "",
   "other": "lm-evaluation-harness's medqa_4options task reads the GBaker/MedQA-USMLE-4-options-hf mirror and scores multiple_choice accuracy and normalized accuracy over the 1,273-question test split.\n"
  },
  "id": "medqa",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "MedQA tests whether a model can pick the correct answer to a clinical multiple-choice question written in the style of the United States Medical Licensing Examination. Most questions present a short patient vignette (age, presenting symptoms, exam findings, sometimes lab values) and ask for a diagnosis, a next step in management, or an underlying mechanism, then offer several candidate answers. It is a single-turn, English-language, text-only task built from real practice-exam question banks rather than written for the benchmark, so it leans on applied clinical reasoning more than isolated fact recall.\n",
  "metric": {
   "baseline_note": "A numeric USMLE passing threshold and a physician baseline score were not established from the sources reviewed for this page, so neither is reported here.\n",
   "direction": "higher_is_better",
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 25,
   "unit": "%"
  },
  "name": "MedQA",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2009.13081",
   "title": "What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams",
   "url": "https://arxiv.org/abs/2009.13081",
   "year": 2020
  },
  "publisher": {
   "authors": [
    "Di Jin",
    "Eileen Pan",
    "Nassim Oufattole",
    "Wei-Hung Weng",
    "Hanyi Fang",
    "Peter Szolovits"
   ],
   "org": "MIT Computer Science and Artificial Intelligence Laboratory (CSAIL)",
   "url": "https://github.com/jind11/MedQA"
  },
  "released": "2020-09",
  "repo_url": "https://github.com/jind11/MedQA",
  "saturation": {
   "as_of": "2025-07",
   "note": "Google's Med-PaLM 2 reported 86.5% on MedQA in May 2023. Google's MedGemma Technical Report put MedGemma 27B at 87.7% zero-shot and 89.8% with best-of-5 test-time scaling in July 2025. No continuously updated public leaderboard was found during this research; recent scores mostly surface inside individual model papers and system cards rather than one standing tracker, so a single current global top score could not be confirmed beyond these two dated data points.\n",
   "status": "watch",
   "top_score": 89.8
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams",
    "url": "https://arxiv.org/abs/2009.13081"
   },
   {
    "accessed": "2026-09-08",
    "title": "jind11/MedQA GitHub repository",
    "url": "https://github.com/jind11/MedQA"
   },
   {
    "accessed": "2026-09-08",
    "title": "GBaker/MedQA-USMLE-4-options-hf dataset card, Hugging Face",
    "url": "https://huggingface.co/datasets/GBaker/MedQA-USMLE-4-options-hf"
   },
   {
    "accessed": "2026-09-08",
    "title": "bigbio/med_qa dataset card, Hugging Face",
    "url": "https://huggingface.co/datasets/bigbio/med_qa"
   },
   {
    "accessed": "2026-09-08",
    "title": "medqa_4options task config, EleutherAI lm-evaluation-harness",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/medqa/medqa.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedGemma Technical Report",
    "url": "https://arxiv.org/abs/2507.05201"
   },
   {
    "accessed": "2026-09-08",
    "title": "Towards Expert-Level Medical Question Answering with Large Language Models (Med-PaLM 2)",
    "url": "https://arxiv.org/abs/2305.09617"
   },
   {
    "accessed": "2026-09-08",
    "title": "google/medgemma-27b-it model card, Hugging Face",
    "url": "https://huggingface.co/google/medgemma-27b-it"
   }
  ],
  "status": "active",
  "subcategory": "medical licensing exam question answering",
  "summary": "Four-option USMLE-style clinical multiple-choice questions, the most widely reported medical exam benchmark for LLMs.",
  "tags": [
   "medical",
   "multiple-choice",
   "usmle",
   "clinical-vignette"
  ],
  "task_format": "Four-option multiple-choice clinical vignette question; the model returns a single letter answer (A-D), usually zero-shot or few-shot.\n"
 }
}