HELM's generation wrap of MEDIQA 2019 Task 3, scoring a free-form answer to a consumer health question with an LLM jury against the expert-ranked gold answer.
unassessed
| Category | domain |
|---|---|
| Subcategory | consumer health question answering |
| Page status | active |
| Metric | medi_qa_accuracy (HELM LLM-jury average of accuracy, completeness and clarity, each 1-5) |
| Direction | higher_is_better |
| Unit | points |
| Dataset size | 150 |
| Dataset licence | CC-BY-4.0 on the authors' GitHub release; Hugging Face bigbio/mediqa_qa lists licence as unknown |
| Publisher | U.S. National Library of Medicine (LHC/NLM) and IBM Research; HELM scenario by Stanford CRFM |
This id is HELM's `medi_qa` scenario, not the 2019 ranking shared task as originally scored, and not MedQA. MEDIQA 2019 Task 3 gave a consumer health question plus CHiQA's retrieved answers and asked systems to filter and re-rank them. HELM instead takes the test questions, uses the expert-ranked number-one answer as the reference, and asks the model to generate an answer from the question alone. Inputs and outputs are English text. The original shared task also had NLI and recognizing-question-entailment tracks; those are not this id.
Zero-shot generation. HELM's prompt is "Answer the following consumer health question." The model writes free text (max 1,024 tokens). An LLM jury then rates the output against the gold answer.
No model card in ModelSpec reports this benchmark yet.