lm-eval's generation wrap of MEDIQA 2019 Task 3: write an English answer to a consumer health question and score overlap against the first listed gold answer.
unassessed
| Category | domain |
|---|---|
| Subcategory | consumer health question answering (generation overlap metrics) |
| Page status | unknown |
| Metric | bleu, rouge1, rouge2, rougeL, bleurt, bert_score (no designated headline) |
| Direction | higher_is_better |
| Dataset size | 150 |
| Dataset licence | CC-BY-4.0 on the authors' GitHub README; Hugging Face bigbio/mediqa_qa lists licence as unknown |
| Publisher | U.S. National Library of Medicine (LHC/NLM) and IBM Research; lm-eval task by EleutherAI |
This id is EleutherAI lm-evaluation-harness task mediqa_qa2019, not the 2019 ranking shared task as originally scored, and not HELM medi_qa. MEDIQA 2019 Task 3 gave a consumer health question plus CHiQA candidate answers and asked systems to filter and re-rank them. lm-eval instead takes the question text and asks the model to generate a free-form answer. Inputs and outputs are English text. The original shared task also had medical NLI and recognizing-question-entailment tracks; those are not this id.
generate_until generation, stopping at a blank line. The YAML description tells the model to answer a patient question as a doctor would. Gold is QUESTION.AnswerList[0].Answer.AnswerText from the Hugging Face source schema, which is XML file order, not the expert's ReferenceRank=1 answer. A separate task mediqa_qa2019_perplexity scores loglikelihood of that gold string.
No model card in ModelSpec reports this benchmark yet.