{
 "body": "\n## What it measures\n\nchw_care_plan is HELM's code name for MedHELM NoteExtract. The model reads a free-form community health worker care-plan note and must fill a fixed English template: chief complaint plus history-of-present-illness slots. The prompt forbids guessing; missing facts should be \"Not mentioned.\" Inputs are English clinical text from a private CSV, not a public Hub dataset.\n\nThis is not [mtsamples_procedures](mtsamples_procedures.md), [mtsamples_replicate](mtsamples_replicate.md), or [mimic_bhc](mimic_bhc.md), which generate plans or hospital-course summaries from other note sources. It is also not a Stanford Health Care `shc_*` scenario.\n\n## How it is scored\n\nThe headline metric is `chw_care_plan_accuracy`, an LLM-jury mean of accuracy, structure, and clarity, each 1\u20135. The paper says NoteExtract has no gold answer, so the jury uses structure instead of completeness. HELM still stores the original \"MO Note\" as the tagged reference and adds BERTScore-style summarization metrics, which assume a gold string those jury axes do not. The default jury is GPT-4o, Llama 3.3 70B Instruct, and Claude 3.7 Sonnet, using Stanford Health Care deployments in the bundled judges file. Missing annotations default to 1.0. Official rows are zero-shot with a 768-token cap. The run-spec docstring calls the task \"summarize doctor-patient dialogues\"; the scenario Python and the paper describe care-plan restructuring. Prefer the scenario and the paper.\n\n## Dataset and licence\n\nOfficial reproductions point at `/share/pi/nigam/datasets/CHW_Dataset.csv` in `run_entries_medhelm_private_stanford.conf`. That path is not in the public MedHELM run-entry file. Rows without an \"MO Note\" are skipped. Table 7 of the MedHELM paper labels the benchmark Private and New under Patient Communication and Education. No n was given in the sources opened for this page. HELM's repository is Apache-2.0; the notes are not.\n\n## Who publishes it\n\nMedHELM is a Stanford CRFM evaluation, with equal first authors Suhana Bedi, Hejie Cui, Miguel Fuentes, and Alyssa Unell on the peer-reviewed paper and Nigam H. Shah as last author. The arXiv preprint (2505.23802) is dated 26 May 2025. Nature Medicine published the article on 20 January 2026 (issue March 2026). CRFM hosts the leaderboard. Nature author metadata lists Nirmal Ravi at eHealth Africa Clinics, Kano, Nigeria; that is an affiliation, not a published collection-site statement for the CSV.\n\n## Lineage\n\nNoteExtract is one of the 18 newly formulated MedHELM benchmarks, not a wrap of an older shared task. It sits with other private clinical-note work in HELM but does not share items with MTSamples or MIMIC-BHC. No successor id exists in this repository.\n\n## Saturation and contamination\n\nThe paper's heatmap discussion names NoteExtract as the strongest-performing benchmark among the 35, which is a reason to treat a high jury score as a weak separator of frontier models. No numeric top score was copied from the live leaderboard. The CSV is private, so training-set contamination of the notes themselves is unlikely; the prompt template is public.\n\n## How to run it\n\nInstall HELM with MedHELM extras and call `helm-run` with spec `chw_care_plan` and a `data_path` to a CSV that has an \"MO Note\" column. Official leaderboard rows use the Stanford private run-entry file and Stanford Health Care model deployments. Anyone else must supply their own path and deployments. There is no lm-evaluation-harness, inspect_evals, or OpenCompass task under this id.\n\n## Reading the numbers\n\nA high NoteExtract jury score means the judges thought the rewrite was accurate, well structured, and clear on these private notes, not that the model can write a billable encounter note in another clinic. Because there is no gold extraction, completeness is not scored; a fluent template that omits facts the note actually contained can still look structured. Do not compare the 1\u20135 jury mean to exact-match medical QA or to BERTScore on the same run. Look at other MedHELM private note tasks and at a harder public clinical-extraction set alongside it. The live leaderboard shows unnormalized metrics; the paper heatmap is normalized for display.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "chw_care_plan",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "NoteExtract",
   "CHW Care Plan",
   "chw_care_plan"
  ],
  "category": "domain",
  "contamination": {
   "note": "The evaluation CSV is private (MedHELM private run-entry file, not the public conf). Pretraining on these notes is unlikely. The prompt template and scoring code are public in HELM.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": null,
   "size_note": "The notes are not public. Official HELM rows load a private CSV at /share/pi/nigam/datasets/CHW_Dataset.csv and skip rows whose \"MO Note\" is NaN. MedHELM Table 7 lists NoteExtract as access Private, curation New, category Patient Communication and Education. No instance count for this CSV was stated in the scenario, the schema, or the ar5iv HTML of arXiv:2505.23802. Nature Medicine author metadata includes Nirmal Ravi at eHealth Africa Clinics, Kano, Nigeria; the paper table does not itself name a country for the notes. HELM code is Apache-2.0; that licence does not cover the CSV.\n",
   "splits": "HELM TEST_SPLIT only, from the caller-supplied CSV",
   "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/chw_care_plan_scenario.py"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-030 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-030"
  },
  "harness": {
   "bigbench": "",
   "helm": "chw_care_plan",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "chw_care_plan",
  "last_updated": "",
  "leaderboard_url": "https://crfm.stanford.edu/helm/medhelm/latest/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "This id is HELM's chw_care_plan scenario, shown on the MedHELM leaderboard as NoteExtract. It is not MTSamples, MIMIC-BHC, or a Stanford Health Care shc_* task. Each item is an English free-form care-plan note in the CSV column \"MO Note\". The prompt asks the model to extract chief complaint and history-of-present-illness fields (onset, provoking/palliating factors, quality, region/radiation, severity, timing, related symptoms) and to write \"Not mentioned\" rather than infer. The paper describes the same work as restructuring community health worker care plans into a specified format without a gold-standard response.\n",
  "metric": {
   "baseline_note": "schema_medhelm.yaml display name is NoteExtract Jury Score. The annotator rates accuracy, structure, and clarity on 1-5 (not completeness). The Nature/arXiv paper says NoteExtract has no gold response, so the jury swapped completeness for structure. HELM still attaches the source \"MO Note\" as the tagged reference and also logs summarization metrics (BERTScore, distilbert-base-uncased). Default jury in helm.benchmark.scenarios.medhelm/judges.yaml: GPT-4o (2024-05-13), Llama 3.3 70B Instruct, Claude 3.7 Sonnet. LLMJuryMetric averages every 1-5 axis score; default_score is 1.0 when annotations are missing. No clinician human baseline for this scenario was published.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 5,
   "name": "chw_care_plan_accuracy (HELM LLM-jury average of accuracy, structure, clarity, each 1-5)",
   "random_baseline": null,
   "unit": "points"
  },
  "name": "NoteExtract (MedHELM chw_care_plan)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2505.23802",
   "title": "Holistic evaluation of large language models for medical tasks with MedHELM",
   "url": "https://www.nature.com/articles/s41591-025-04151-2",
   "year": 2026
  },
  "publisher": {
   "authors": [
    "Suhana Bedi",
    "Hejie Cui",
    "Miguel Fuentes",
    "Alyssa Unell",
    "Michael Wornow",
    "Yifan Mai",
    "Percy Liang",
    "Nigam H. Shah"
   ],
   "org": "Stanford CRFM / MedHELM",
   "url": "https://crfm.stanford.edu/helm/medhelm/latest/"
  },
  "released": "2025",
  "repo_url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/chw_care_plan_scenario.py",
  "saturation": {
   "as_of": "",
   "note": "The MedHELM paper says the Figure 3 heatmap's strongest cells are on NoteExtract, so the task may not separate frontier models as sharply as harder MedHELM sets. No numeric top jury score was read from the leaderboard or the paper tables opened here.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM CHWCarePlanScenario (MO Note, OPQRST-style prompt, scenario name chw_care_plan)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/chw_care_plan_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "CHWCarePlanAnnotator (accuracy, structure, clarity on 1-5)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/annotation/chw_care_plan_annotator.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "get_chw_care_plan_run_spec (zero-shot, max_tokens 768, jury metric)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/run_specs/medhelm_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "schema_medhelm.yaml (NoteExtract display name, chw_care_plan_accuracy)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/static/schema_medhelm.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "Default MedHELM jury models",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/medhelm/judges.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "Private Stanford run entries (CHW_Dataset.csv; not in the public conf)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/presentation/run_entries_medhelm_private_stanford.conf"
   },
   {
    "accessed": "2026-09-08",
    "title": "LLMJuryMetric (mean of 1-5 axis scores)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/metrics/llm_jury_metrics.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM MedHELM docs (public / gated / private run-entry files)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/docs/medhelm.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "stanford-crfm/helm Apache License 2.0",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM preprint (arXiv:2505.23802, submitted 2025-05-26)",
    "url": "https://arxiv.org/abs/2505.23802"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM full text (Table 7 NoteExtract private/new; no-gold jury; heatmap note)",
    "url": "https://ar5iv.labs.arxiv.org/html/2505.23802"
   },
   {
    "accessed": "2026-09-08",
    "title": "Nature Medicine article (published 2026-01-20; Nirmal Ravi affiliation Kano, Nigeria)",
    "url": "https://www.nature.com/articles/s41591-025-04151-2"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM leaderboard",
    "url": "https://crfm.stanford.edu/helm/medhelm/latest/"
   }
  ],
  "status": "active",
  "subcategory": "structured extraction from community-health-worker care-plan notes",
  "summary": "MedHELM private NoteExtract task: rewrite a community health worker care-plan note into a fixed clinical template, scored by an LLM jury.",
  "tags": [
   "medhelm",
   "clinical",
   "note-extraction",
   "private-data",
   "llm-jury"
  ],
  "task_format": "Zero-shot generation. HELM adapter instructions: \"Follow the instructions provided regarding conversion of a patient note into a specified format.\" Empty input and output nouns. max_train_instances=0, max_tokens=768. Official private run entries pass data_path=/share/pi/nigam/datasets/CHW_Dataset.csv.\n"
 }
}