{
 "body": "\n## What it measures\n\nSTARR Patient Instructions, shown as \"PatientInstruct\" in MedHELM's own taxonomy, gives a model real\nclinical case details -- a diagnosis, the procedure performed, and the history-and-physical and\noperative notes -- and asks it to generate clear, actionable instructions for a patient recovering\nfrom that procedure. It falls under MedHELM's \"Patient Communication and Education\" category and\ntargets clinician-facing-to-patient-facing text synthesis: turning clinical documentation into\ninstructions a patient can actually follow, not diagnosis or treatment-planning reasoning.\n\n## How it is scored\n\nHELM combines two kinds of metric. Automatic summarization metrics (a BERTScore-based comparison\nagainst the real discharge instructions on file) run alongside an LLM-as-judge score named\n`starr_patient_instructions_accuracy`, computed by HELM's LLMJuryMetric, which uses a panel of\nannotator language models to rate generated instructions rather than relying on surface overlap with\nthe reference text alone. No numeric ceiling, random baseline or human baseline for this specific\nmetric was found in the sources read for this page.\n\n## Dataset and licence\n\nThe scenario is built from Stanford's STARR-OMOP clinical data repository, covering after-visit\ninstructions for outpatient surgeries and procedures with same-day (within 24 hours) discharge. Each\ninstance requires five fields -- Diagnosis, ActualProcedure, HistoryPhysicalNoteText,\nOperativeNoteText and DischargeInstructionNoteText -- and only records flagged QC=\"TRUE\" are used.\nThe MedHELM paper lists this benchmark as newly created for the study with \"Private\" access: it\ncannot be shared outside Stanford's institutional data use agreements and patient privacy\nprotections, so no public URL, licence or exact item count was found for this page.\n\n## Who publishes it\n\nThe scenario was built by the Stanford Center for Research on Foundation Models (CRFM) in\npartnership with Stanford Health Care, as part of the MedHELM evaluation suite described in Bedi et\nal., \"MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks\" (2025), a\nmulti-institution collaboration with 81 listed authors including Percy Liang and Nigam H. Shah. HELM\nmaintains the code and documentation for running it.\n\n## Lineage\n\nThis scenario has no predecessor, successor or variant tracked in this repository. The MedHELM paper\nreports 13 newly developed benchmarks and 14 benchmarks with private access (STARR Patient\nInstructions is one of both, per its \"New\"/\"Private\" listing), out of 35 benchmarks total spanning\nMedHELM's five task categories; the paper does not specify how many of the 14 private benchmarks\ncame from the Stanford Health Care partnership specifically versus other private sources. None of\nMedHELM's other benchmarks have pages in this repository yet.\n\n## Saturation and contamination\n\nNo per-benchmark leaderboard score for this specific scenario was found in the sources read for this\npage; the MedHELM paper's headline results are reported in aggregate across its full 35-benchmark\nsuite (advanced reasoning models led, with Claude 3.5 Sonnet reported as competitive at lower\nestimated cost), not broken out per task in the material read, so saturation status is not\nestablished. Contamination risk is low: because the data is a private, real-patient clinical\nresource restricted by institutional data use agreements, it is not part of any public web crawl and\ncannot appear in general pretraining corpora, though this also means independent verification of\nresults is only possible within Stanford's own evaluation setup.\n\n## How to run it\n\nImplemented in HELM as the `starr_patient_instructions` scenario\n(`helm/benchmark/scenarios/starr_patient_instructions_scenario.py`), paired with the\n`StarrPatientInstructionsAnnotator` for the LLM-jury scoring step. Because the underlying clinical\ndata is private, the scenario cannot be run outside Stanford's own HELM deployment with access to\nthe STARR-OMOP source files; no other harness (lm-evaluation-harness, inspect_evals, OpenCompass,\nBIG-bench) implementation exists for it.\n\n## Reading the numbers\n\nA strong score here suggests a model can turn dense clinical documentation into instructions a real\npatient could plausibly follow after a procedure, a practically important but narrow skill distinct\nfrom clinical reasoning or diagnosis. Because scoring blends an LLM-jury judgment with automatic\nsummarization metrics, results depend on which models serve as judges, a choice that is not\nindependently confirmed from the sources read for this page; because the dataset is private and\nresults are not independently reproducible outside Stanford's infrastructure, treat any score on this\nbenchmark as reported, not independently verifiable.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "starr_patient_instructions",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "PatientInstruct"
  ],
  "category": "domain",
  "contamination": {
   "note": "The dataset is drawn from real, private Stanford Health Care patient records under institutional data use agreements and is explicitly not publicly shareable, so it cannot appear in web-scale pretraining corpora the way public benchmarks can. This also means results cannot be independently reproduced or audited outside Stanford's own evaluation infrastructure.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Not public; restricted under Stanford Health Care institutional data use agreements and patient privacy protections, per the MedHELM paper",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": null,
   "size_note": "Exact instance count is not published; the MedHELM paper lists this as a \"New\" benchmark with \"Private\" access, built from Stanford's STARR-OMOP clinical data repository, covering after-visit instructions for outpatient surgeries and procedures with discharge within 24 hours. Records are filtered to those with a QC=\"TRUE\" quality-control flag before use. The dataset cannot be shared outside Stanford's institutional data use agreements because it contains real patient records, so an item count was not confirmed from a public source for this page.\n",
   "splits": "All instances assigned to the test split (zero-shot); no public train/validation split",
   "url": ""
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Claude Sonnet 5, sonnet-batch-004 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, sonnet-batch-004"
  },
  "harness": {
   "bigbench": "",
   "helm": "starr_patient_instructions",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "starr_patient_instructions",
  "last_updated": "",
  "leaderboard_url": "https://crfm-helm.readthedocs.io/en/latest/medhelm/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "STARR Patient Instructions (displayed as \"PatientInstruct\" in MedHELM's taxonomy) gives a model real-world clinical case details -- a diagnosis, the planned or performed procedure, and the history-and-physical and operative notes -- and asks it to generate clear, actionable post-procedure instructions appropriate for a patient recovering from that intervention. It sits in MedHELM's \"Patient Communication and Education\" category and tests clinical note synthesis and patient-facing communication, not diagnostic or decision-support reasoning.\n",
  "metric": {
   "baseline_note": "Not established from the sources read for this page; the MedHELM paper's public tables do not give a specific numeric baseline or ceiling for this metric in the material read.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "LLMJuryMetric ('starr_patient_instructions_accuracy', an LLM-as-judge score from a jury of annotator models) combined with automatic summarization metrics (BERTScore-based) via HELM's summarization metric suite",
   "random_baseline": null,
   "unit": "score"
  },
  "name": "STARR Patient Instructions (PatientInstruct)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2505.23802",
   "title": "MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks",
   "url": "https://arxiv.org/abs/2505.23802",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Suhana Bedi",
    "Hejie Cui",
    "Miguel Fuentes"
   ],
   "org": "Stanford Center for Research on Foundation Models (CRFM), in partnership with Stanford Health Care",
   "url": "https://crfm-helm.readthedocs.io/en/latest/medhelm/"
  },
  "released": "2025-05",
  "repo_url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/starr_patient_instructions_scenario.py",
  "saturation": {
   "as_of": "",
   "note": "Not established from the sources read for this page; the MedHELM paper reports aggregate findings across its 35-benchmark suite (advanced reasoning models scoring highest overall, with Claude 3.5 Sonnet offering comparable results at lower estimated cost) but no per-benchmark score for this specific scenario was found in the material read.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM starr_patient_instructions_scenario.py source (scenario description, STARR-OMOP data source, five required fields, QC filter, test-only split)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/starr_patient_instructions_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM MedHELM schema (display name PatientInstruct, taxonomy: text generation / clinician / post-procedure, metric list including starr_patient_instructions_accuracy)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/static/schema_medhelm.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "Bedi et al., 'MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks' (arXiv abstract: authors, 5 categories/22 subcategories/121 tasks/35 benchmarks, 9 LLMs evaluated)",
    "url": "https://arxiv.org/abs/2505.23802"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM paper HTML (Table 7/8: PatientInstruct listed under Patient Communication & Education, curation status New, access Private)",
    "url": "https://arxiv.org/html/2505.23802v2"
   }
  ],
  "status": "active",
  "subcategory": "clinical patient communication: generate personalized post-procedure discharge instructions from real clinical case data",
  "summary": "A MedHELM scenario, built from private Stanford Health Care records, that asks a model to write post-procedure patient instructions from a diagnosis, procedure and clinical notes.",
  "tags": [
   "clinical",
   "medical",
   "patient-communication",
   "generation",
   "private-dataset",
   "helm"
  ],
  "task_format": "Text generation, zero-shot (all instances are assigned to the test split; no in-context training examples are drawn from the data itself). Each instance is built from a record with five fields -- Diagnosis, ActualProcedure, HistoryPhysicalNoteText, OperativeNoteText and DischargeInstructionNoteText -- filtered to records marked QC=\"TRUE\"; the model receives the first four as input and its output is compared against the real DischargeInstructionNoteText as reference.\n"
 }
}