{
 "body": "\n## What it measures\n\nMIMIC-RRS, in MedHELM, is radiology report summarization: given the\nFindings section, write the Impression. Chen, Varma, Wan, Langlotz and\nDelbrouck (ACL 2023) built MIMIC-RRS so the task would cover more than\nchest X-ray \u2014 CT, MR and X-ray, and anatomies including head, spine,\nabdomen and pelvis. HELM's scenario restricts itself to MIMIC-III files\nand does not load MIMIC-CXR.\n\n## How it is scored\n\nHELM is zero-shot with a 128-token cap. The named metric\n`mimic_rrs_accuracy` averages LLM-jury scores for accuracy, completeness\nand clarity (1\u20135) against the gold Impression. Summarization overlap\nmetrics run in parallel. Chen et al. instead reported lexical overlap and\na factuality metric on their paper splits. A HELM jury point and a 2023\nROUGE number are not the same evaluation.\n\n## Dataset and licence\n\nChen et al. Table 2: 207,782 pairs total \u2014 79,779 from MIMIC-III and\n128,003 from MIMIC-CXR. HELM's module docstring says 73,259 MIMIC-III\nreports. That 73,259 vs 79,779 disagreement is unresolved here; neither\nfigure is treated as HELM's test size, which is the length of the local\n`test.*.tok` files and is unpublished. Source notes are MIMIC-III\n(PhysioNet credentialed). Chen et al. release reconstruction code and\nsplits rather than a fully public labelled corpus.\n\n## Who publishes it\n\nThe dataset paper is ACL 2023 (arXiv 2211.08584, November 2022). Zhihong\nChen and Xiang Wan are at CUHK Shenzhen; Maya Varma, Curtis Langlotz and\nJean-Benoit Delbrouck are at Stanford AIMI. MedHELM packaging and the\n`mimic_rrs` scenario are Stanford CRFM.\n\n## Lineage\n\nNot an alias of [mimic_repsum](mimic_repsum.md). That lm-eval task uses\n`dmacres/mimiciii-hospitalcourse-meta` and ROUGE/BLEU/RadGraph, not Chen\net al.'s tokenized files and not HELM's jury. Related MedHELM generation:\n[mimic_bhc](mimic_bhc.md) (discharge BHC) and [ACI-Bench](aci_bench.md).\n\n## Saturation and contamination\n\nNo public HELM top score is recorded. Contamination is low for the\nofficial credentialed files; Chen et al. already warned that some\nMIMIC-pretrained clinical models overlap this test set.\n\n## How to run it\n\nPhysioNet access to MIMIC-III, reconstruct or obtain the tokenized\nfindings/impression files, then `helm-run` scenario `mimic_rrs` with\n`data_path` to that directory. Gated MedHELM only. Match jury models\nbefore comparing `mimic_rrs_accuracy`.\n\n## Reading the numbers\n\nA high jury score means annotator models thought the Impression captured\nthe Findings. It does not replace a radiologist's sign-off, and it is\neasy to confuse with other MIMIC \"report summarization\" harnesses that\nuse different files and metrics. Check whether a number is HELM jury,\nChen et al. overlap, or [mimic_repsum](mimic_repsum.md) RadGraph before\nranking models.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mimic_rrs",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "MIMIC RRS",
   "Radiology Report Summarization"
  ],
  "category": "domain",
  "contamination": {
   "note": "MIMIC-III requires PhysioNet credentialing. HELM lists mimic_rrs only in run_entries_medhelm_gated.conf. Chen et al. note that some clinical LMs were pretrained on MIMIC-III overlapping this set and therefore excluded those models from their paper experiments.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "PhysioNet credentialed access for MIMIC-III source notes; Chen et al. release reconstruction scripts, not a public labelled dump",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": null,
   "size_note": "Chen et al. Table 2: 207,782 reports in full MIMIC-RRS (79,779 MIMIC-III across 11 modality-anatomy pairs plus 128,003 MIMIC-CXR chest X-ray). HELM's docstring says it uses only MIMIC-III and quotes 73,259 reports \u2014 that does not match the paper's 79,779 MIMIC-III count, and this page records both rather than choosing one. HELM reads only the test split files (test.findings.tok / test.impression.tok); the HELM test cardinality is not published. Gated paths use /share/pi/nigam/data/rrs-mimiciii/all.\n",
   "splits": "Chen et al. put CT abdomen/pelvis, CT chest, CT neck, CT spine, CT head, MR head and X-ray chest in train/validation/test and hold MR pelvis/spine/neck, MR abdomen and CT sinus as OOD tests; HELM uses the test files only",
   "url": "https://arxiv.org/abs/2211.08584"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-006 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-006"
  },
  "harness": {
   "bigbench": "",
   "helm": "mimic_rrs",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "MedHELM gated; helm-run needs data_path to a directory with test.findings.tok and test.impression.tok"
  },
  "id": "mimic_rrs",
  "last_updated": "",
  "leaderboard_url": "https://crfm.stanford.edu/helm/medhelm/latest",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "mimic_bhc",
    "mimic_repsum"
   ]
  },
  "measures": "This id is HELM's mimic_rrs scenario. Chen et al. (ACL 2023) released MIMIC-RRS as findings\u2013impression pairs from MIMIC-III and MIMIC-CXR across CT, MR and X-ray and several anatomies. HELM's scenario file states it uses only the MIMIC-III reports, loading test.findings.tok and test.impression.tok from a local directory. The model writes an Impression from Findings. MedHELM's main score is an LLM jury (mimic_rrs_accuracy); overlap metrics are logged as well. English clinical text.\n",
  "metric": {
   "baseline_note": "Schema display name is MIMIC-RRS Jury Score. HELM also runs summarization metrics (BERTScore distilbert-base-uncased and related overlap scores). Chen et al. reported ROUGE-style and factuality metrics on their own splits, not this 1\u20135 jury. No HELM jury human baseline is published.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 5,
   "name": "mimic_rrs_accuracy (HELM LLM-jury average of accuracy, completeness, clarity, each 1-5)",
   "random_baseline": null,
   "unit": "points"
  },
  "name": "MIMIC-RRS (MedHELM)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2211.08584",
   "title": "Toward Expanding the Scope of Radiology Report Summarization to Multiple Anatomies and Modalities",
   "url": "https://aclanthology.org/2023.acl-short.41/",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Zhihong Chen",
    "Maya Varma",
    "Xiang Wan",
    "Curtis Langlotz",
    "Jean-Benoit Delbrouck"
   ],
   "org": "Stanford University (AIMI / CRFM MedHELM packaging); The Chinese University of Hong Kong, Shenzhen (Chen, Wan)",
   "url": "https://crfm.stanford.edu/helm/medhelm/latest"
  },
  "released": "2023",
  "repo_url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/mimic_rrs_scenario.py",
  "saturation": {
   "as_of": "",
   "note": "MedHELM reports the gated task, but no static top jury score was captured from public HTML. Chen et al.'s own ROUGE-style tables are a different metric and a different (paper) split.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM mimic_rrs_scenario.py (MIMIC-III only; 73,259 in docstring)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/mimic_rrs_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM get_mimic_rrs_spec",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/run_specs/medhelm_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM MIMICRRSAnnotator jury prompt",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/annotation/mimic_rrs_annotator.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "schema_medhelm.yaml MIMIC-RRS",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/static/schema_medhelm.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM gated run entries (rrs-mimiciii/all)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/presentation/run_entries_medhelm_gated.conf"
   },
   {
    "accessed": "2026-09-08",
    "title": "Chen et al. arXiv 2211.08584 (79,779 MIMIC-III + 128,003 MIMIC-CXR)",
    "url": "https://arxiv.org/abs/2211.08584"
   },
   {
    "accessed": "2026-09-08",
    "title": "ACL 2023 short paper",
    "url": "https://aclanthology.org/2023.acl-short.41/"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM access levels",
    "url": "https://github.com/stanford-crfm/helm/blob/main/docs/medhelm.md"
   }
  ],
  "status": "active",
  "subcategory": "radiology findings-to-impression summarization",
  "summary": "MedHELM's gated wrap of MIMIC-RRS on MIMIC-III: generate an Impression from Findings, scored by an LLM jury plus overlap metrics.",
  "tags": [
   "biomedical",
   "radiology",
   "summarization",
   "medhelm",
   "gated",
   "llm-jury"
  ],
  "task_format": "Zero-shot generation. HELM: \"Generate the impression section of the radiology report based on its findings. This will not be used to diagnose nor treat any patients. Be as concise as possible.\" Input Findings, output Impression, max_tokens=128, max_train_instances=0.\n"
 }
}