An NIH biomedical-research QA benchmark of 68,227 expert- and template-generated questions on neurodegenerative-disease genetics, molecular biology and clinical knowledge, LLM-judged for quality and safety.
unassessed
| Category | domain |
|---|---|
| Subcategory | biomedical research question answering (neurodegenerative-disease pilot domain) |
| Page status | active |
| Metric | LLM-judged correctness; the paper's own two-axis Response Quality Rate and Safety Rate |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 68227 |
| Dataset licence | OpenRAIL++, per the Hugging Face dataset card's own metadata -- an unusual choice for a text QA dataset (more common for model weights); no separate licence statement was found in the paper itself. |
| Publisher | Center for Alzheimer's and Related Dementias (CARD), National Institute on Aging, National Institutes of Health, with DataTecnica LLC, the Department of Computer Science at Johns Hopkins University, and NIA's Laboratory of Neurogenetics |
CARDBiomedBench tests whether a model can answer complex biomedical-research questions that require integrating genetic, molecular and clinical knowledge, rather than simple fact lookup. Its pilot implementation focuses on neurodegenerative diseases (NDDs), a domain the authors chose because it demands combining several kinds of specialised evidence. Items were built by combining expert-annotated question-answer pairs with semi-automated, template-based augmentation drawn from authoritative public resources: drug-development data, genome-wide association studies (GWAS), and summary-data-based Mendelian randomisation (SMR) analyses. Each item is tagged with one or more of ten biological categories (for example, "Drug Disease Relations" or "Drug Gene Relations") and one or more of nine reasoning-skill categories (for example, "Multi-Filter" or "Join," names that echo the database-query templates the augmented items were generated from). It is a single-turn, English-only, text-only, open-ended question-answering task.
Open-ended free-text biomedical question, answered zero-shot with no answer options offered; the free-text response is graded against a reference answer by a separate LLM judge rather than by exact string match, since correct biomedical answers vary in phrasing.
No model card in ModelSpec reports this benchmark yet.