MedHELM's gated wrap of MIMIC-RRS on MIMIC-III: generate an Impression from Findings, scored by an LLM jury plus overlap metrics.
unassessed
| Category | domain |
|---|---|
| Subcategory | radiology findings-to-impression summarization |
| Page status | active |
| Metric | mimic_rrs_accuracy (HELM LLM-jury average of accuracy, completeness, clarity, each 1-5) |
| Direction | higher_is_better |
| Unit | points |
| Dataset licence | PhysioNet credentialed access for MIMIC-III source notes; Chen et al. release reconstruction scripts, not a public labelled dump |
| Publisher | Stanford University (AIMI / CRFM MedHELM packaging); The Chinese University of Hong Kong, Shenzhen (Chen, Wan) |
This id is HELM's mimic_rrs scenario. Chen et al. (ACL 2023) released MIMIC-RRS as findings–impression pairs from MIMIC-III and MIMIC-CXR across CT, MR and X-ray and several anatomies. HELM's scenario file states it uses only the MIMIC-III reports, loading test.findings.tok and test.impression.tok from a local directory. The model writes an Impression from Findings. MedHELM's main score is an LLM jury (mimic_rrs_accuracy); overlap metrics are logged as well. English clinical text.
Zero-shot generation. HELM: "Generate the impression section of the radiology report based on its findings. This will not be used to diagnose nor treat any patients. Be as concise as possible." Input Findings, output Impression, max_tokens=128, max_train_instances=0.
No model card in ModelSpec reports this benchmark yet.