308 USMLE Step 2/3-style clinical vignette questions with expert explanations, built specifically to be harder than MedQA and to test explanation quality, not just the answer letter.
unassessed
| Category | domain |
|---|---|
| Subcategory | USMLE Step 2/3-style clinical multiple-choice questions with expert explanations |
| Page status | active |
| Metric | accuracy (plus separate explanation-quality evaluation) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 308 |
| Dataset licence | Not stated: the GitHub repository carries no licence file or badge, and the underlying question bank is republished from the Medbullets USMLE-preparation website rather than authored by the paper's authors. |
| Publisher | Rice University and Johns Hopkins University |
Medbullets presents a clinical vignette in the style of USMLE Step 2 and Step 3 (later-stage, more applied medical-licensing exam content than Step 1) and asks the model to choose the correct diagnosis or management step from four or five answer options. Each item also carries an expert-written explanation, because the paper that introduced Medbullets was built specifically to let researchers evaluate the quality of a model's reasoning, not only whether it picked the right letter. The underlying questions are drawn from the existing Medbullets USMLE-preparation question bank rather than written for the paper, so they reflect realistic, simulated clinical scenarios rather than deliberately adversarial ones.
Four-option ("op4") or five-option ("op5") multiple-choice clinical vignette question; the model returns a single letter answer, typically zero- or few-shot, sometimes paired with a free-text explanation that is separately evaluated.
No model card in ModelSpec reports this benchmark yet.