Four-option questions that ask a model to pick the correct description of an ICD or ATC medical code among related distractors, across three difficulty levels.
unassessed
| Category | domain |
|---|---|
| Subcategory | medical coding concept multiple-choice |
| Page status | active |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 819832 |
| Dataset licence | Apache-2.0 |
| Publisher | Ben-Gurion University of the Negev |
MedConceptsQA tests whether a model can map a medical code to its official description. Each item names one code from ICD-9-CM or ICD-10-CM (diagnoses), ICD-9-PCS or ICD-10-PCS (procedures), or ATC (drugs), then offers four English descriptions. Only one description matches the given code. Distractors are sampled from the same vocabulary at a controlled graph distance, so hard items pit a code against near neighbours that share a parent. It is a single-turn, English, text-only multiple-choice task about code literacy rather than clinical vignettes or exam reasoning, which sets it apart from MedQA and MedMCQA.
Four-option multiple choice. The model returns a letter A-D. EleutherAI's lm-evaluation-harness uses four-shot by default (first_n from the dev split) and also supports zero-shot.
No model card in ModelSpec reports this benchmark yet.