{
 "body": "\n## What it measures\n\nMIMIC-BHC, as MedHELM runs it, is discharge-note summarization. The model\nreads an English clinical note derived from MIMIC-IV and must write the\nBrief Hospital Course \u2014 the narrative of the stay that clinicians actually\nfile. Aali et al. built MIMIC-IV-BHC by stripping, sectioning and pairing\nnotes with their BHC targets. HELM's prompt is a single instruction:\nsummarize the clinical note into a brief hospital course.\n\n## How it is scored\n\nHELM is zero-shot (`max_train_instances=0`). The headline metric\n`mimic_bhc_accuracy` is an LLM jury: annotator models score accuracy,\ncompleteness and clarity from 1 to 5 against the gold BHC\n(`MIMICBHCAnnotator`). HELM still computes summarization overlap\n(BERTScore and related metrics) on the same outputs. The 2024 paper's\nBLEU/BERT-Score tables and the five-clinician preference study on 30\nnotes are a different protocol. Do not put a JAMIA BLEU next to a HELM\njury point.\n\n## Dataset and licence\n\nPhysioNet project MIMIC-IV-Ext-BHC\n(`labelled-notes-hospital-course` v1.2.0, published 3 February 2025)\nand the paper both give 270,033 pairs. Licence is PhysioNet Credentialed\nHealth Data License 1.5.0 with DUA 1.5.0 and CITI training. HELM does\nnot ship the notes; gated run entries pass a local `mimic_iv_bhc.json`.\nThe paper separately samples 2,000/100 splits inside 0\u20131,024, 1,024\u20132,048\nand 2,048\u20134,096 token bins for its own experiments.\n\n## Who publishes it\n\nDataset and JAMIA 2024 paper: Asad Aali and colleagues at Stanford\n(doi:10.1093/jamia/ocae312; arXiv 2403.05720, March 2024). Reference\ncode: github.com/StanfordMIMI/clin-bhc-summ. The HELM scenario and\nMedHELM leaderboard are Stanford CRFM.\n\n## Lineage\n\nNo predecessor page in this repository. Related MedHELM clinical\ngeneration tasks: [MIMIC-RRS](mimic_rrs.md) (radiology\nfindings\u2192impression on MIMIC-III) and [ACI-Bench](aci_bench.md)\n(dialogue to visit note). [mimic_repsum](mimic_repsum.md) is a separate\nlm-eval findings\u2192impression wrap over a MIMIC-III hospital-course Hugging\nFace dump, not this BHC task.\n\n## Saturation and contamination\n\nNo public top jury score is recorded here. Contamination is graded low\nfor the official labelled files because they sit behind PhysioNet\ncredentialing and HELM's gated run list.\n\n## How to run it\n\nCredentialed PhysioNet access, then `helm-run` with scenario `mimic_bhc`\nand `data_path` to the local JSON. MedHELM docs class this as gated\n(`run_entries_medhelm_gated.conf`). Jury models come from\n`jury_config_path`. Without the same jury and the same local file, scores\nare not comparable.\n\n## Reading the numbers\n\nA high HELM jury score means annotator models judged the generated BHC\naccurate, complete and clear relative to the gold section. It is not a\nclaim that a physician would sign the note, and it is not the paper's\nBLEU number. The gold BHC is itself a clinician-written section of the\nsame stay, so overlap can reward style matching. Look at [ACI-Bench](aci_bench.md)\nor [HealthBench](healthbench.md) for other clinical generation settings,\nand never treat a public-web \"MIMIC summarization\" score as this gated\ntask.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mimic_bhc",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "MIMIC-IV-BHC",
   "MIMIC-IV-Ext-BHC",
   "Brief Hospital Course"
  ],
  "category": "domain",
  "contamination": {
   "note": "Official files are PhysioNet credentialed (access policy: DUA plus CITI Data or Specimens Only Research). HELM registers the scenario in run_entries_medhelm_gated.conf, not the public run-entry file. Web-scale crawls are unlikely to contain this labelled dump. Underlying MIMIC-IV notes still appear in credentialed research use.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "PhysioNet Credentialed Health Data License 1.5.0 (DUA 1.5.0; CITI training required)",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": 270033,
   "size_note": "PhysioNet MIMIC-IV-Ext-BHC v1.2.0 and the JAMIA/arXiv paper both state 270,033 note\u2013BHC pairs (mean input 2,267\u00b1914 tokens, mean BHC 564\u00b1410). Source notes: MIMIC-IV-Note, 331,794 discharge summaries from 145,915 patients (paper). The paper's own modelling used 2,000/100 train/test draws inside three context-length bins, not the full 270,033. HELM comments out train/validate, reads a local JSON/JSONL of input/target fields, and keeps only TEST_SPLIT; the row count of that HELM file is not published. MedHELM gated runs point at /share/pi/nigam/data/bhc-mimiciv/mimic_iv_bhc.json.\n",
   "splits": "PhysioNet corpus 270,033 pairs; paper bins 2,000/100 per context range; HELM test-only from a local file of unpublished size",
   "url": "https://physionet.org/content/labelled-notes-hospital-course/1.2.0/"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-006 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-006"
  },
  "harness": {
   "bigbench": "",
   "helm": "mimic_bhc",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "MedHELM gated; helm-run needs data_path to a local MIMIC-IV-BHC JSON/JSONL"
  },
  "id": "mimic_bhc",
  "last_updated": "2025-02",
  "leaderboard_url": "https://crfm.stanford.edu/helm/medhelm/latest",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "mimic_rrs",
    "mimic_repsum"
   ]
  },
  "measures": "This id is HELM's mimic_bhc scenario, not the authors' original BLEU and BERTScore study by itself. MIMIC-IV-BHC pairs a preprocessed MIMIC-IV discharge note with the Brief Hospital Course section of that stay. HELM prompts the model to summarize the note into a BHC (zero-shot, max 1,024 tokens) and grades the text with an LLM jury on accuracy, completeness and clarity (1\u20135) as mimic_bhc_accuracy, while also logging summarization overlap metrics. Inputs and outputs are English clinical text.\n",
  "metric": {
   "baseline_note": "HELM's main metric is the jury score (schema_medhelm.yaml display name MIMIC-BHC Jury Score). HELM also attaches summarization metrics (BERTScore with distilbert-base-uncased, and related overlap scores). The 2024 JAMIA paper instead reports BLEU and BERT-Score plus a five- clinician reader study on 30 notes. Those figures are not on the HELM 1\u20135 jury scale. No HELM jury human baseline was published.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 5,
   "name": "mimic_bhc_accuracy (HELM LLM-jury average of accuracy, completeness, clarity, each 1-5)",
   "random_baseline": null,
   "unit": "points"
  },
  "name": "MIMIC-BHC (MedHELM)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2403.05720",
   "title": "A dataset and benchmark for hospital course summarization with adapted large language models",
   "url": "https://doi.org/10.1093/jamia/ocae312",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Asad Aali",
    "Dave Van Veen",
    "Yamin Ishraq Arefeen",
    "Jason Hom",
    "Christian Bluethgen",
    "Eduardo Pontes Reis",
    "Sergios Gatidis",
    "Namuun Clifford",
    "Joseph Daws",
    "Arash S. Tehrani",
    "Jangwon Kim",
    "Akshay S. Chaudhari"
   ],
   "org": "Stanford University (MIMI / CRFM MedHELM packaging)",
   "url": "https://crfm.stanford.edu/helm/medhelm/latest"
  },
  "released": "2024-03",
  "repo_url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/mimic_bhc_scenario.py",
  "saturation": {
   "as_of": "",
   "note": "MedHELM lists mimic_bhc on the gated healthcare leaderboard. Public HTML for that board did not yield a static top jury score here, so saturation is unknown. The paper's GPT-4 ICL vs fine-tuned Llama2-13B comparison is on BLEU/BERT-Score and a 30-note reader study, not HELM jury points.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM mimic_bhc_scenario.py (270,033 notes; test-only JSONL)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/mimic_bhc_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM get_mimic_bhc_spec (zero-shot, LLM jury)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/run_specs/medhelm_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "schema_medhelm.yaml MIMIC-BHC Jury Score",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/static/schema_medhelm.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM access levels (gated = PhysioNet)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/docs/medhelm.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM gated run entries (mimic_iv_bhc.json path)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/presentation/run_entries_medhelm_gated.conf"
   },
   {
    "accessed": "2026-09-08",
    "title": "Aali et al. hospital course summarization (arXiv 2403.05720)",
    "url": "https://arxiv.org/abs/2403.05720"
   },
   {
    "accessed": "2026-09-08",
    "title": "JAMIA 2024 paper (doi:10.1093/jamia/ocae312)",
    "url": "https://doi.org/10.1093/jamia/ocae312"
   },
   {
    "accessed": "2026-09-08",
    "title": "PhysioNet MIMIC-IV-Ext-BHC v1.2.0 (published 2025-02-03)",
    "url": "https://physionet.org/content/labelled-notes-hospital-course/1.2.0/"
   },
   {
    "accessed": "2026-09-08",
    "title": "StanfordMIMI/clin-bhc-summ (PhysioNet DOI 10.13026/fh2q-4148)",
    "url": "https://github.com/StanfordMIMI/clin-bhc-summ"
   }
  ],
  "status": "active",
  "subcategory": "discharge-note summarization into a Brief Hospital Course",
  "summary": "MedHELM's gated wrap of MIMIC-IV-BHC: write a Brief Hospital Course from a discharge note, scored by an LLM jury plus overlap metrics.",
  "tags": [
   "biomedical",
   "clinical-notes",
   "summarization",
   "medhelm",
   "gated",
   "llm-jury"
  ],
  "task_format": "Zero-shot generation. HELM instructions: \"Summarize the clinical note into a brief hospital course.\" Input noun Clinical Note, output noun Brief Hospital Course. max_train_instances=0, max_tokens=1024.\n"
 }
}