lm-eval's generation wrap of BI55/MedText: write a diagnosis and treatment plan from an English patient presentation and score overlap metrics.
unassessed
| Category | domain |
|---|---|
| Subcategory | synthetic patient presentation to diagnosis and treatment |
| Page status | unknown |
| Metric | bleu, rouge1, rouge2, rougeL, bleurt, bert_score (no designated headline) |
| Direction | higher_is_better |
| Dataset size | 1412 |
| Dataset licence | CC-BY-4.0 |
| Publisher | Hugging Face user BI55 (card fullname L. Heinrich) |
This id is EleutherAI lm-evaluation-harness task medtext. The Hugging Face set BI55/MedText pairs an English patient presentation (Prompt) with a diagnosis and treatment completion written in a clinical voice. lm-eval asks the model to continue that presentation. It is text-only English generation. It is not a licensing exam, not MIMIC notes, and not the 2019 Melamud and Shivade synthetic clinical-note paper that the lm-eval README cites.
generate_until generation, stopping at a blank line. doc_to_text is the Prompt field; gold is Completion. The YAML description instructs the model to answer as a doctor with a likely diagnosis and treatment. Train, validation and test all point at the single train split. Companion task medtext_perplexity scores rolling likelihood of the completion.
No model card in ModelSpec reports this benchmark yet.