{
 "body": "\n## What it measures\n\nMedConceptsQA asks a model to identify the official description of a medical code. Each question\nnames one code from ICD-9-CM, ICD-10-CM, ICD-9-PCS, ICD-10-PCS or ATC, then lists four English\ndescriptions. Only one description is correct. Easy distractors are drawn from anywhere in the same\nvocabulary. Medium and hard distractors come from codes a few graph edges away, so hard items look\nalike. The task is English text, single-turn, and about code meaning rather than a patient vignette.\n\nThat focus is the point. Clinical notes often store diagnoses, procedures and drugs as codes, not\nprose. A model that scores well on MedQA can still fail here if it cannot tell neighbouring ICD\nstrings apart.\n\n## How it is scored\n\nThe metric is four-option accuracy. Chance is 25%, because the correct letter is placed at random.\nThe paper evaluates each vocabulary and difficulty with zero-shot and four-shot prompts, sampling\n250 items and repeating three times for a 95% confidence interval. Aggregates average those runs.\nEleutherAI's group `med_concepts_qa` instead averages accuracy across the five vocabulary groups,\neach of which already averages its easy, medium and hard tasks. The authors' own harness command\nuses `--limit 250`, so a number from a full 819k-item run is not comparable to the published tables.\n\n## Dataset and licence\n\nHugging Face reports 819,832 examples on the `all` config (60 dev, 819,772 test), matching the\npaper's per-cell counts once the 60 dev items are included. Fifteen configs split the five\nvocabularies by easy, medium and hard. The Hugging Face card states Apache-2.0. Answers ship with\nthe files. The authors also published the work in *Computers in Biology and Medicine* (2024).\n\n## Who publishes it\n\nOfir Ben Shoham and Nadav Rappoport at Ben-Gurion University of the Negev introduced the benchmark\nin May 2024 (arXiv:2405.07348). Code lives at nadavlab/MedConceptsQA. That README is the standing\nleaderboard; it invites submissions by GitHub issue. No separate hosted leaderboard was found.\n\n## Lineage\n\nThis is not a variant of MedQA, MedMCQA or PubMedQA. Those pages cover exam vignettes or abstract\nyes/no/maybe questions. MedConceptsQA is generated from PyHealth vocabulary graphs. No successor\nid exists in this repository.\n\n## Saturation and contamination\n\nThe authors' README still lists gpt-4-0125-preview first, at 52.489% zero-shot and 61.911%\nfour-shot, matching the paper. Several 70B models land in the high 40s to high 50s few-shot.\nClinical models in the paper sat near chance. The ceiling is not in reach. Contamination risk is\nmedium: the labelled set has been public since May 2024, but items are generated from public code\ntables rather than copied from exams.\n\n## How to run it\n\nIn lm-evaluation-harness, run group `med_concepts_qa`, or a single task such as\n`med_concepts_qa_icd10cm_hard`. The template scores multiple-choice accuracy on ofir408/MedConceptsQA\nwith four-shot from the four-item dev split unless you set `--num_fewshot 0`. The authors' README\nuses `--limit 250`. Compare shot count, limit and whether the figure is one vocabulary or the\nfive-group mean before lining numbers up.\n\n## Reading the numbers\n\nA high score means the model can pick the right gloss for a billing or drug code among close\nneighbours, not that it can diagnose or prescribe. The paper's own case study shows GPT-4 dropping\nfrom the mid-90s on easy ICD-9-CM to the low 50s on hard items, so an aggregate hides that slide.\nLook at MedQA or MedMCQA for vignette reasoning, and treat any MedConceptsQA figure that does not\nstate shot count and sample size as incomparable.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "med_concepts_qa",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "MedConceptsQA",
   "medconceptsqa"
  ],
  "category": "domain",
  "contamination": {
   "note": "Questions are generated from public medical-code vocabularies and have been downloadable with answers on Hugging Face since May 2024. They are not copied from licensing exams, but the full labelled set is ungated.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Apache-2.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 819832,
   "size_note": "Hugging Face config `all` reports 60 dev and 819,772 test examples (819,832 total). That total matches the paper's per-vocabulary table once the 60 later-held-out dev items are included. Each of the 15 vocabulary-by-difficulty configs holds 4 dev items; test counts run from 4,434 (icd9proc_hard) to 190,983 (icd10proc easy and medium). The paper's published evaluation samples 250 items per vocabulary, difficulty and shot setting rather than scoring the full test set.\n",
   "splits": "60 dev / 819,772 test on the `all` config; 15 vocabulary-by-difficulty configs with 4 dev items each",
   "url": "https://huggingface.co/datasets/ofir408/MedConceptsQA"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch 7 pilot (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, pilot-review"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "med_concepts_qa",
   "opencompass": "",
   "other": "Group `med_concepts_qa` averages accuracy across five vocabulary groups (icd9cm, icd10cm, icd9proc, icd10proc, atc). Each group is the tag of three tasks (`easy`, `medium`, `hard`), for 15 runnable task names such as `med_concepts_qa_atc_easy`. Dataset path is ofir408/MedConceptsQA. The authors' README evaluates with `--limit 250` and `--num_fewshot` 0 or 4.\n"
  },
  "id": "med_concepts_qa",
  "last_updated": "",
  "leaderboard_url": "https://github.com/nadavlab/MedConceptsQA",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "MedConceptsQA tests whether a model can map a medical code to its official description. Each item names one code from ICD-9-CM or ICD-10-CM (diagnoses), ICD-9-PCS or ICD-10-PCS (procedures), or ATC (drugs), then offers four English descriptions. Only one description matches the given code. Distractors are sampled from the same vocabulary at a controlled graph distance, so hard items pit a code against near neighbours that share a parent. It is a single-turn, English, text-only multiple-choice task about code literacy rather than clinical vignettes or exam reasoning, which sets it apart from MedQA and MedMCQA.\n",
  "metric": {
   "baseline_note": "Four balanced options, with the correct letter placed at random, so uniform guessing is 25%. No human baseline was reported in the paper or on the authors' GitHub leaderboard.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 25,
   "unit": "%"
  },
  "name": "MedConceptsQA",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2405.07348",
   "title": "MedConceptsQA: Open Source Medical Concepts QA Benchmark",
   "url": "https://arxiv.org/abs/2405.07348",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Ofir Ben Shoham",
    "Nadav Rappoport"
   ],
   "org": "Ben-Gurion University of the Negev",
   "url": "https://github.com/nadavlab/MedConceptsQA"
  },
  "released": "2024-05",
  "repo_url": "https://github.com/nadavlab/MedConceptsQA",
  "saturation": {
   "as_of": "2024-05",
   "note": "The authors' GitHub README leaderboard, which matches the paper's GPT-4 figures, puts gpt-4-0125-preview at 52.489% zero-shot and 61.911% four-shot, both well below 100% and well above the 25% chance rate. Several 70B medical and general models sit in the high 40s to high 50s few-shot. No later dated public tracker was found during this research.\n",
   "status": "open",
   "top_score": 61.911
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "MedConceptsQA: Open Source Medical Concepts QA Benchmark",
    "url": "https://arxiv.org/abs/2405.07348"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedConceptsQA HTML full text on ar5iv",
    "url": "https://ar5iv.labs.arxiv.org/html/2405.07348"
   },
   {
    "accessed": "2026-09-08",
    "title": "ofir408/MedConceptsQA dataset card, Hugging Face",
    "url": "https://huggingface.co/datasets/ofir408/MedConceptsQA"
   },
   {
    "accessed": "2026-09-08",
    "title": "ofir408/MedConceptsQA Hugging Face API",
    "url": "https://huggingface.co/api/datasets/ofir408/MedConceptsQA"
   },
   {
    "accessed": "2026-09-08",
    "title": "ofir408/MedConceptsQA datasets-server split counts",
    "url": "https://datasets-server.huggingface.co/info?dataset=ofir408/MedConceptsQA"
   },
   {
    "accessed": "2026-09-08",
    "title": "nadavlab/MedConceptsQA GitHub repository",
    "url": "https://github.com/nadavlab/MedConceptsQA"
   },
   {
    "accessed": "2026-09-08",
    "title": "med_concepts_qa task README, EleutherAI lm-evaluation-harness",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/med_concepts_qa/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "med_concepts_qa group config, EleutherAI lm-evaluation-harness",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/med_concepts_qa/_med_concepts_qa.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "med_concepts_qa default template, EleutherAI lm-evaluation-harness",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/med_concepts_qa/_default_template_yaml"
   }
  ],
  "status": "active",
  "subcategory": "medical coding concept multiple-choice",
  "summary": "Four-option questions that ask a model to pick the correct description of an ICD or ATC medical code among related distractors, across three difficulty levels.",
  "tags": [
   "medical",
   "multiple-choice",
   "medical-coding",
   "icd",
   "atc"
  ],
  "task_format": "Four-option multiple choice. The model returns a letter A-D. EleutherAI's lm-evaluation-harness uses four-shot by default (first_n from the dev split) and also supports zero-shot.\n"
 }
}