{
 "body": "\n## What it measures\n\nBMT-Status asks a model to read a real English hematology or oncology note and\nanswer a yes-or-no question about transplant status. HELM's scenario text says\nthe notes concern bone marrow transplant, hematopoietic stem cell transplant, or\nhematopoietic cell transplant, and that the target is whether the patient later\nreceived another transplant. The class docstring adds that the gold labels come\nfrom post-transplant status questions, not from a public exam set.\n\nThe input is text only. HELM concatenates the CSV question and the note into one\nprompt and forbids any answer other than A (yes) or B (no). This is a closed\nchart-review item, not free-text generation and not a USMLE-style knowledge\nquestion.\n\n## How it is scored\n\nHELM scores the task with exact match of the generated letter to the gold label,\nand also logs quasi-exact and prefix exact-match variants. The schema names\nexact_match on the test split as the main metric. Scores are a fraction in\n0-1. The Nature Medicine suite heatmap normalizes closed-ended scores to the\nsame 0-1 scale for display; the live leaderboard is described as showing the\nunnormalized metric.\n\nNo published random or human baseline for this private set was found. Because\nthe labels are A or B, a coin-flip number would only be valid if the two classes\nwere equally common, which was not reported.\n\n## Dataset and licence\n\nNature Medicine Extended Data Table 1 lists 220 instances evaluated, access\nlevel private, curation status new. HELM loads a caller-supplied CSV with\ncolumns prompt, context, and label, and the Stanford private run file points at\nmedhelm-BMT-dataset_filtered.csv. Every instance is tagged as the test split.\nThe scenario does not publish a train split.\n\nThe notes are Stanford Health Care EHR text. A table note says \"instances\nevaluated\" is the subset used for scoring, capped near 1,000 for larger sets.\nThe paper says 14 datasets cannot be shared because of institutional agreements\nand patient privacy; that group also includes non-Stanford private sets such as\nNoteExtract (Nigeria) and MentalHealth (India). No licence string for the CSV\nwas found. HELM's evaluation code is Apache 2.0; that licence does not cover\nthe private notes.\n\n## Who publishes it\n\nThe task sits in MedHELM, a Stanford CRFM and Stanford Health Care collaboration\nwith Microsoft Health and Life Sciences, coordinated by the Center for\nBiomedical Informatics Research. Equal first authors on the peer-reviewed paper\nare Suhana Bedi, Hejie Cui, Miguel Fuentes, and Alyssa Unell. The Nature\nMedicine article lists Bedi as corresponding author; Nigam H. Shah is last\nauthor. The preprint (arXiv:2505.23802) appeared 26 May 2025; the\nNature Medicine article is dated 20 January 2026 (issue March 2026). The HELM\nscenario file entered the public repository with MedHELM V1 on 19 March 2025.\nCRFM hosts the leaderboard.\n\n## Lineage\n\nBMT-Status is one of the new private Stanford Health Care tasks added so\nMedHELM could cover hospital work that public exam sets miss. In the published\ntaxonomy it sits under Medical Research Assistance, subcategory Recording\nresearch processes, not under Administration. It is not an alias of MedQA,\nPubMedQA, or any other public transplant quiz.\n\nSibling Stanford Health Care HELM scenarios in this repository include shc_cdi\n(CDI-QA), shc_conf (MedConfInfo), shc_ent (ENT-Referral), shc_gip\n(HospiceReferral), shc_privacy (PrivacyDetection), shc_proxy (ProxySender),\nshc_ptbm (ADHD-Behavior), shc_sei (ADHD-MedEffects), and shc_sequoia\n(ClinicReferral). There is no MedHELM family page.\n\n## Saturation and contamination\n\nSaturation is not established. MedHELM reports overall win rates and a heatmap\nacross the suite, but this page could not read a numeric BMT-Status exact-match\nfrom that figure. Contamination risk is low relative to public exam sets: the\nauthors keep these notes private specifically to reduce training-data inclusion\nand to test generalization beyond public medical benchmarks. The 14 private\nMedHELM datasets are not all from Stanford Health Care.\n\n## How to run it\n\nThe reference run spec is `shc_bmt_med` in stanford-crfm/helm. It requires a\nlocal `data_path` to the private CSV. Official Stanford runs live in\n`run_entries_medhelm_private_stanford.conf`. The adapter is joint\nmultiple-choice with instructions \"Answer A or B.\" HELM's multiple-choice helper\ndefaults to five in-context examples, but this scenario emits only a test split,\nso the in-context sampler has no train instances and the run is effectively\nzero-shot.\n\nOutsiders cannot reproduce the official numbers without the CSV. MedHELM's docs\nsay private-set scores are produced inside Stanford's secure environment;\nexternal models can be submitted for that run via the HELM repository. No\nlm-evaluation-harness, Inspect Evals, OpenCompass, or BIG-bench task with this\nitem set was confirmed.\n\n## Reading the numbers\n\nA high exact-match score means the model usually picked the gold yes or no on\nthese 220 transplant-status items under HELM's single-letter protocol. It does\nnot mean the model is safe to use for transplant registry work: the notes are\nfrom one health system, the labels are not public, and the metric ignores\ncalibrated uncertainty. Compare it with other private Stanford referral and\nstatus tasks in the same suite, not with MedQA accuracy. If two reports disagree,\ncheck that both used `shc_bmt_med` exact match and the same private file.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "shc_bmt",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "shc_bmt_med",
   "BMT-Status"
  ],
  "category": "domain",
  "contamination": {
   "note": "Nature Medicine lists BMT-Status among 14 private datasets that cannot be shared because of institutional agreements and patient privacy, and says private sets are held out to limit training inclusion. Those 14 are not all Stanford Health Care records.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": 220,
   "size_note": "Nature Medicine Extended Data Table 1 reports 220 instances evaluated. That is the published evaluation count, not a counted CSV. The file is a private filtered CSV (medhelm-BMT-dataset_filtered.csv) with prompt, context, and label columns. HELM tags every loaded row as test.\n",
   "splits": "test only (HELM TEST_SPLIT); no published train or validation split",
   "url": ""
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-001 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-001"
  },
  "harness": {
   "bigbench": "",
   "helm": "shc_bmt_med",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "shc_bmt",
  "last_updated": "2025-08",
  "leaderboard_url": "https://crfm.stanford.edu/helm/medhelm/latest/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "shc_cdi",
    "shc_conf",
    "shc_ent",
    "shc_gip",
    "shc_privacy",
    "shc_proxy",
    "shc_ptbm",
    "shc_sei",
    "shc_sequoia"
   ]
  },
  "measures": "BMT-Status tests whether a model can read an English clinical note from a hematology or oncology consultation and answer a binary question about bone marrow transplant (BMT), hematopoietic stem cell transplant (HSCT), or hematopoietic cell transplant (HCT) status. HELM's published goal is to decide whether the patient received a subsequent transplant from the documentation alone. The harness id is shc_bmt; the runnable HELM scenario is shc_bmt_med.\n",
  "metric": {
   "baseline_note": "HELM's schema lists exact_match on the test split as the main metric, with quasi and prefix exact-match variants also recorded. Label balance and a human rater baseline were not published for this private set.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1.0,
   "name": "exact_match",
   "random_baseline": null,
   "unit": "0-1 scale"
  },
  "name": "BMT-Status",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2505.23802",
   "title": "Holistic evaluation of large language models for medical tasks with MedHELM",
   "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC13267972/",
   "year": 2026
  },
  "publisher": {
   "authors": [
    "Suhana Bedi",
    "Hejie Cui",
    "Miguel Fuentes",
    "Alyssa Unell",
    "Nigam H. Shah"
   ],
   "org": "Stanford Health Care and Stanford CRFM (MedHELM)",
   "url": "https://crfm.stanford.edu/helm/medhelm/latest/"
  },
  "released": "2025-03",
  "repo_url": "https://github.com/stanford-crfm/helm",
  "saturation": {
   "as_of": "",
   "note": "MedHELM publishes a per-benchmark heatmap, but this page did not recover a dated numeric top exact-match for BMT-Status from that figure or from a parsed leaderboard table.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM shc_bmt_scenario.py (SHCBMTMedScenario, BMT-Status)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/shc_bmt_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM medhelm_run_specs.py (shc_bmt_med run spec)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/run_specs/medhelm_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM run_entries_medhelm_private_stanford.conf",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/presentation/run_entries_medhelm_private_stanford.conf"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM schema_medhelm.yaml (BMT-Status group)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/static/schema_medhelm.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks (arXiv:2505.23802)",
    "url": "https://arxiv.org/abs/2505.23802"
   },
   {
    "accessed": "2026-09-08",
    "title": "Holistic evaluation of large language models for medical tasks with MedHELM (PMC author manuscript)",
    "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC13267972/"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM documentation (access levels and private run entries)",
    "url": "https://crfm-helm.readthedocs.io/en/latest/medhelm/"
   },
   {
    "accessed": "2026-09-08",
    "title": "stanford-crfm/helm Apache License 2.0",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/LICENSE"
   }
  ],
  "status": "active",
  "subcategory": "bone marrow transplant status from clinical notes",
  "summary": "Private Stanford Health Care MedHELM task: given a hematology note, answer yes or no on whether the patient later received a bone marrow or stem-cell transplant.\n",
  "tags": [
   "medical",
   "clinical",
   "classification",
   "private-dataset",
   "medhelm",
   "stanford-health-care",
   "transplant"
  ],
  "task_format": "Joint multiple-choice generation. HELM wraps a CSV prompt and note as one English string and requires a single token, A for yes or B for no, with no extra text.\n"
 }
}