lm-eval task mimic_repsum: write an Impression from Findings parsed out of a MIMIC-III hospital-course dump, scored with ROUGE, BLEU, BERTScore, BLEURT and RadGraph-F1.
unassessed
| Category | domain |
|---|---|
| Subcategory | findings-to-impression summarization over MIMIC-III notes in lm-evaluation-harness |
| Page status | active |
| Metric | F1-Radgraph plus BLEU, ROUGE-1/2/L, BLEURT and BERTScore (nanmean) |
| Direction | higher_is_better |
| Dataset size | 5356 |
| Publisher | EleutherAI (harness task); Hugging Face user dmacres (dataset dump); MIMIC-III from MIT Lab for Computational Physiology |
mimic_repsum is EleutherAI lm-evaluation-harness's radiology-style findings-to-impression task. The YAML loads Hugging Face dmacres/mimiciii-hospitalcourse-meta (train 24,993, validation 5,356, test 5,356). For each row, utils.py searches extractive_notes_summ for FINDING and IMPRESSION headings and prompts "Given the findings: {}. Summarize the findings." The README calls this MIMIC-III Report Summarization and cites the 2016 MIMIC-III database paper, not a dedicated summarization paper. It is not MedHELM's mimic_rrs.
generate_until. Prompt from parsed findings; target is the parsed impression. Decoding stops at a blank line; top_p=0.95. A sibling task mimic_repsum_perplexity scores perplexity on the same setup.
No model card in ModelSpec reports this benchmark yet.