{
 "body": "\n## What it measures\n\nlm-eval `mimic_repsum` asks a model to turn radiology Findings into an\nImpression. The prompt is built by regex over the `extractive_notes_summ`\nfield of `dmacres/mimiciii-hospitalcourse-meta`: the harness finds\nFINDING and IMPRESSION headings, feeds the findings span, and uses the\nimpression span as the reference. The Hub dataset is named as a hospital-\ncourse collection; the task code is nonetheless a findings\u2192impression\nsummarizer, which is why the README's title is MIMIC-III Report\nSummarization.\n\nThis is not [mimic_rrs](mimic_rrs.md). MedHELM loads Chen et al.'s\ntokenized MIMIC-RRS files and scores an LLM jury. It is also not\n[mimic_bhc](mimic_bhc.md), which writes Brief Hospital Course from\nMIMIC-IV discharge notes.\n\n## How it is scored\n\nDefault `mimic_repsum` is `generate_until` with `until: [\"\\n\\n\"]` and\n`top_p: 0.95`. `process_results` computes BLEU, ROUGE-1/2/L, BLEURT\n(bleurt-base-512), BERTScore F1, and F1-Radgraph (partial reward).\nAggregations are `nanmean`. Very short targets or predictions become\nNaN rather than zero. `mimic_repsum_perplexity` is a separate task on\nthe same documents. There is no single headline metric in the YAML.\n\n## Dataset and licence\n\nHub `dataset_info` gives 24,993 / 5,356 / 5,356 rows. Features include\nsubject_id, hadm_id, target_text, extractive_notes_summ, n_notes and a\nlist of notes. The card has no licence; tags do not include a licence\neither. Underlying MIMIC-III is PhysioNet credentialed (Johnson et al.,\nScientific Data 2016). Whether the dmacres dump is a permitted public\nredistribution is not established from the card.\n\n## Who publishes it\n\nThe runnable evaluation is EleutherAI's harness task (YAML metadata\nversion 1.4). The dataset repository is Hugging Face user `dmacres`\n(created 15 November 2023). The README's bibliographic citation is the\nMIMIC-III database paper, not Chen et al. 2023 and not a dmacres paper \u2014\nno separate summarization paper for this packaging was found.\n\n## Lineage\n\nDo not fold this id into [mimic_rrs](mimic_rrs.md) or\n[mimic_bhc](mimic_bhc.md). Shared ancestry is MIMIC clinical text.\nThe closest task shape is findings\u2192impression, which MIMIC-RRS also\nuses, with different files, splits, licence and metrics.\n\n## Saturation and contamination\n\nSaturation unknown. Contamination is high for the Hub dump: note text\nis public there. That is the opposite access model from MedHELM's gated\nPhysioNet scenarios.\n\n## How to run it\n\n`lm_eval --tasks mimic_repsum` (and optionally `mimic_repsum_perplexity`).\nInstall the extra metrics the utils import (`evaluate`, `bert-score`,\n`rouge_score`, `radgraph`, BLEURT from Google's repo). Compare only to\nother lm-eval runs of this YAML version.\n\n## Reading the numbers\n\nA high ROUGE or RadGraph-F1 here means the model's impression overlapped\nthe parsed gold span on this Hub dump. It does not measure MedHELM jury\nquality, does not use Chen et al.'s official MIMIC-RRS splits, and the\nheading parser can feed messy spans when FINDING/IMPRESSION order\nvaries. Prefer [mimic_rrs](mimic_rrs.md) when the claim is about\nMIMIC-RRS, and treat this score as harness-specific.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mimic_repsum",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "domain",
  "contamination": {
   "note": "The Hugging Face dump is public (created 2023-11-15 on the Hub API) with note text and extractive summaries in the clear. MIMIC-III itself is a 2016 PhysioNet credentialed corpus; this mirror's redistribution rights are not stated on the card. Models trained after late 2023 could have seen the Hub files.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 5356,
   "size_note": "Hugging Face dmacres/mimiciii-hospitalcourse-meta: train 24,993, validation 5,356, test 5,356 (Hub API dataset_info, accessed 2026-09-08). lm-eval binds training_split, validation_split and test_split to those three names. After heading parse, some test rows may be skipped as NaN; that filtered count was not measured. The dataset card has no licence field.\n",
   "splits": "train 24,993 / validation 5,356 / test 5,356 on the Hub; lm-eval uses all three names",
   "url": "https://huggingface.co/datasets/dmacres/mimiciii-hospitalcourse-meta"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-006 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-006"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "mimic_repsum",
   "opencompass": "",
   "other": "Sibling lm-eval task mimic_repsum_perplexity; YAML version 1.4"
  },
  "id": "mimic_repsum",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "mimic_rrs",
    "mimic_bhc"
   ]
  },
  "measures": "mimic_repsum is EleutherAI lm-evaluation-harness's radiology-style findings-to-impression task. The YAML loads Hugging Face dmacres/mimiciii-hospitalcourse-meta (train 24,993, validation 5,356, test 5,356). For each row, utils.py searches extractive_notes_summ for FINDING and IMPRESSION headings and prompts \"Given the findings: {}. Summarize the findings.\" The README calls this MIMIC-III Report Summarization and cites the 2016 MIMIC-III database paper, not a dedicated summarization paper. It is not MedHELM's mimic_rrs.\n",
  "metric": {
   "baseline_note": "The YAML lists seven metrics; none is declared the single official score. F1-Radgraph uses the radgraph package at reward_level=partial (README links Delbrouck et al., Patterns 2023). Rows with impression or prediction shorter than 5 characters are scored as NaN. No human baseline is given in the task files.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "F1-Radgraph plus BLEU, ROUGE-1/2/L, BLEURT and BERTScore (nanmean)",
   "random_baseline": null,
   "unit": ""
  },
  "name": "MIMIC-III Report Summarization (lm-eval)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "MIMIC-III, a freely accessible critical care database",
   "url": "https://www.nature.com/articles/sdata201635",
   "year": 2016
  },
  "publisher": {
   "authors": [],
   "org": "EleutherAI (harness task); Hugging Face user dmacres (dataset dump); MIMIC-III from MIT Lab for Computational Physiology",
   "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/mimic_repsum"
  },
  "released": "2023-11",
  "repo_url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/mimic_repsum",
  "saturation": {
   "as_of": "",
   "note": "No public leaderboard for the lm-eval task was found. Do not read MedHELM mimic_rrs jury scores as this metric set.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "lm-eval mimic_repsum README",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/mimic_repsum/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval mimic_repsum.yaml (task name, dataset_path, metrics)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/mimic_repsum/mimic_repsum.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval mimic_repsum utils.py (FINDING/IMPRESSION parse)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/mimic_repsum/utils.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "dmacres/mimiciii-hospitalcourse-meta Hub API (split sizes; no licence tag)",
    "url": "https://huggingface.co/api/datasets/dmacres/mimiciii-hospitalcourse-meta"
   },
   {
    "accessed": "2026-09-08",
    "title": "Johnson et al. 2016 MIMIC-III database paper (cited by the task README)",
    "url": "https://www.nature.com/articles/sdata201635"
   },
   {
    "accessed": "2026-09-08",
    "title": "RadGraph-F1 paper linked from the task README",
    "url": "https://www.cell.com/patterns/fulltext/S2666-3899(23)00157-5"
   }
  ],
  "status": "active",
  "subcategory": "findings-to-impression summarization over MIMIC-III notes in lm-evaluation-harness",
  "summary": "lm-eval task mimic_repsum: write an Impression from Findings parsed out of a MIMIC-III hospital-course dump, scored with ROUGE, BLEU, BERTScore, BLEURT and RadGraph-F1.",
  "tags": [
   "biomedical",
   "radiology",
   "summarization",
   "lm-eval",
   "mimic-iii"
  ],
  "task_format": "generate_until. Prompt from parsed findings; target is the parsed impression. Decoding stops at a blank line; top_p=0.95. A sibling task mimic_repsum_perplexity scores perplexity on the same setup.\n"
 }
}