{
 "body": "\n## What it measures\n\nMedAlign evaluates instruction following grounded in longitudinal electronic health records. Each example contains a clinician-authored instruction or question and a patient record represented as an event stream. The expected response is a clinician-generated completion.\n\nThe task exercises medical reading comprehension, record grounding, and clinically informed reasoning in English.\n\n## How it is scored\n\nThe scenario documentation names COMET and BERTScore. HELM\u2019s metadata calls the main metric `medalign_accuracy`, but the scenario does not specify the exact metric aggregation or calibration. Reports should identify the active evaluator and model version.\n\n## Dataset and licence\n\nHELM loads a dataframe through a helper and emits test instances; the scenario code itself does not state row counts. The paper reports 983 natural-language instructions curated by 15 clinicians across 7 specialties, of which 303 have clinician-written reference responses grounded in 276 longitudinal EHRs. The authors distribute MedAlign under a research data use agreement rather than as an open download, so access requires a qualified-researcher request; a redistribution licence was not established.\n\n## Who publishes it\n\nFleming and collaborators introduced MedAlign in the paper \u201cMedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records\u201d (arXiv:2308.14089). HELM maintains an integration scenario. No current standalone leaderboard was established.\n\n## Lineage\n\nMedAlign is a standalone EHR-grounded benchmark. The sources read do not establish a predecessor, successor, or formal variant.\n\n## Saturation and contamination\n\nSaturation is unknown. Contamination risk is assessed as low because the authors distribute MedAlign under a research data use agreement rather than an open download, limiting how the records could reach public training corpora; the inspected sources do not otherwise establish training exposure. A high similarity score can also reward wording overlap without validating clinical safety.\n\n## How to run it\n\nUse HELM\u2019s `medalign` scenario with its configured `max_length` and data path. It wraps each prompt and clinician response as a test instance. Record truncation, record serialization, and whether protected data access was authorized.\n\n## Reading the numbers\n\nA strong score suggests that a model can follow the tested instructions while using supplied longitudinal context. It does not establish diagnostic correctness, safe treatment advice, or robustness to missing or contradictory records. Review factual grounding and clinician assessment alongside automated metrics.\n\nLongitudinal records can contain repeated events, temporal ordering, and irrelevant details. Truncation or serialization changes may alter the available evidence, so those choices should be treated as part of the protocol.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "medalign",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "instruction-following",
  "contamination": {
   "note": "The paper distributes MedAlign under a research data use agreement rather than an open download, which limits how the records could enter public training corpora; actual training exposure and a formal contamination analysis were not established.",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": 303,
   "size_note": "The paper reports 983 natural-language instructions in total, of which 303 have clinician-written reference responses grounded in 276 longitudinal EHRs; the HELM scenario itself does not restate these counts.",
   "splits": "test",
   "url": "https://arxiv.org/abs/2308.14089"
  },
  "freshness": {
   "luna-batch-057 (Codex coordinated)": null,
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, luna-batch-057"
  },
  "harness": {
   "bigbench": "",
   "helm": "medalign",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "medalign",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "MedAlign gives a model an instruction or question grounded in an event-stream style patient record. It evaluates whether the model can read the record and produce the clinician-generated completion.",
  "metric": {
   "baseline_note": "The HELM scenario names these metrics; their active calibration is not stated there.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "COMET and BERTScore",
   "random_baseline": null,
   "unit": ""
  },
  "name": "MedAlign",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2308.14089",
   "title": "MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records",
   "url": "https://arxiv.org/abs/2308.14089",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Scott L. Fleming",
    "Alejandro Lozano",
    "William J. Haberkorn",
    "Jenelle A. Jindal",
    "Eduardo P. Reis",
    "Rahul Thapa",
    "Louis Blankemeier",
    "Julian Z. Genkins",
    "Ethan Steinberg",
    "Ashwin Nayak",
    "Birju S. Patel",
    "Chia-Chun Chiang",
    "Alison Callahan",
    "Zepeng Huo",
    "Sergios Gatidis",
    "Scott J. Adams",
    "Oluseyi Fayanju",
    "Shreya J. Shah",
    "Thomas Savage",
    "Ethan Goh",
    "Akshay S. Chaudhari",
    "Nima Aghaeepour",
    "Christopher Sharp",
    "Michael A. Pfeffer",
    "Percy Liang",
    "Jonathan H. Chen",
    "Keith E. Morse",
    "Emma P. Brunskill",
    "Jason A. Fries",
    "Nigam H. Shah"
   ],
   "org": "Stanford Medicine and collaborators",
   "url": "https://arxiv.org/abs/2308.14089"
  },
  "released": "2023",
  "repo_url": "https://arxiv.org/abs/2308.14089",
  "saturation": {
   "as_of": "",
   "note": "No current authoritative leaderboard was established.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM MedAlign scenario",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/medalign_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedAlign paper",
    "url": "https://arxiv.org/abs/2308.14089"
   }
  ],
  "status": "active",
  "subcategory": "electronic health records",
  "summary": "MedAlign tests instruction following grounded in longitudinal electronic health records and clinician responses.",
  "tags": [
   "medical",
   "ehr",
   "instruction-following"
  ],
  "task_format": "EHR-grounded natural-language instruction and generated clinical response."
 }
}