The first manually constructed radiology visual question answering dataset, pairing clinician questions about head, chest and abdominal images with reference answers.
unassessed
| Category | multimodal |
|---|---|
| Subcategory | medical visual question answering (radiology) |
| Page status | active |
| Metric | Accuracy (exact match for closed/yes-no questions; model-graded semantic match for open-ended questions) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 2244 |
| Dataset licence | CC BY 4.0 (paper); the Hugging Face dataset card states CC0 1.0 Universal, a discrepancy not resolved by a source read for this page |
| Publisher | Lister Hill National Center for Biomedical Communications, U.S. National Library of Medicine |
VQA-RAD shows a model a radiology image (a head CT/MRI, chest X-ray, or abdominal CT) together with a naturally occurring clinical question about it, such as its imaging modality, anatomical plane, organ system, or whether an abnormality is present, and asks the model to answer. It measures whether a model can combine basic radiology image interpretation with reading comprehension of a clinically phrased question, across 11 question categories spanning both yes/no (closed) and free-text (open) answer types.
The model receives an image and a question in a single multimodal turn and must answer in text. inspect_evals' implementation scores closed-ended (yes/no) questions by exact string match and open-ended questions with a tool-calling grader model that judges semantic equivalence to the reference answer, then reports closed, open, and overall accuracy.
No model card in ModelSpec reports this benchmark yet.