{
 "body": "\n## What it measures\n\nHELM `mtsamples_replicate` asks a model to write an English treatment plan from\na mixed-specialty transcribed report. Sources include clinic notes, consults\nand some procedure write-ups from MTSamples.com. HELM hides the PLAN section\nwhen it exists. If there is no PLAN, it still uses SUMMARY or FINDINGS as the\nreference but leaves those sections in the prompt.\n\nThis is MedHELM's \"MTSamples\" decision-support task. It is not\n[mtsamples_procedures](mtsamples_procedures.md), which is the operative-note\nsubset with a stricter strip of all three headers.\n\n## How it is scored\n\nMain metric `mtsamples_replicate_accuracy` is a three-judge 1-5 mean on\naccuracy, completeness and clarity of the proposed plan. Overlap metrics such\nas BERTScore are secondary. Decoding is zero-shot, 512 tokens. Missing jury\noutput becomes 1.0. The annotator prompt tells judges to use history,\nmedications and symptoms and to compare with the gold plan.\n\n## Dataset and licence\n\nThe scenario pins `ebc104a` and downloads `mtsamples_processed`. That folder\nhas 1,269 `.txt` files. The post-filter instance count is not published. No\nSPDX licence is stated. Nature Medicine treats the benchmark as public. The\nwebsite named in HELM metadata is mtsamples.com.\n\n## Who publishes it\n\nStanford CRFM MedHELM (Bedi et al., arXiv 2505.23802; Nature Medicine 2026)\ndefines the scenario and jury. The text is MTSamples samples mirrored on\nGitHub. The board is crfm.stanford.edu/helm/medhelm.\n\n## Lineage\n\nSibling [mtsamples_procedures](mtsamples_procedures.md) is the surgical folder\nand a note-generation group in the same schema. Other MedHELM generation ids\nin this repository include [ACI-Bench](aci_bench.md) and [MIMIC-BHC](mimic_bhc.md).\nThere is no original MTSamples shared-task paper.\n\n## Saturation and contamination\n\nNo scenario-level top score is recorded here. The notes have been public for\nyears, so contamination is high. A strong jury number on this dump is weak\nevidence of private-EHR treatment planning.\n\n## How to run it\n\n`helm-run` with `mtsamples_replicate`. Name the jury models. Do not average\nwith `mtsamples_procedures` or with exact-match MedQA.\n\n## Reading the numbers\n\nA high score means the jury liked the plan relative to a short PLAN/SUMMARY\nline from a public sample note. It does not measure drug-safety checking or\nguideline citations. Many notes are educational transcriptions, not complete\ncharts. Use gated EHR tasks when the claim is real-clinic planning.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mtsamples_replicate",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "MTSamples",
   "mtsamples_processed"
  ],
  "category": "domain",
  "contamination": {
   "note": "Mixed-specialty MTSamples pages and the GitHub mirror are public. Nature Medicine lists MTSamples among unrestricted public datasets. Reference PLAN text is in the same files.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "At HELM pin ebc104a4f96c5b7602242f301e081e9934a23344 the Git tree has 1,269 .txt files under mtsamples_processed. HELM skips files with none of PLAN, SUMMARY or FINDINGS, so scored n is at most 1,269 and was not counted here. A sampled Angina note uses a TREATMENT header rather than PLAN, which this extractor would miss. The live MTSamples.com homepage (2026-09-08) advertises 5,043 samples in 40 specialties; HELM uses a processed GitHub subset, not that full catalogue. MedHELM lists the benchmark as public and reformulated. No SPDX licence in HELM or the data README. The site allows educational print/share with credit and says samples are user-contributed and not guaranteed complete.\n",
   "splits": "HELM TEST_SPLIT only, from filtered GitHub .txt files",
   "url": "https://github.com/raulista1997/benchmarkdata/tree/ebc104a4f96c5b7602242f301e081e9934a23344/mtsamples_processed"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-013 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-013"
  },
  "harness": {
   "bigbench": "",
   "helm": "mtsamples_replicate",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "MedHELM run spec mtsamples_replicate; main_metric mtsamples_replicate_accuracy, main_split test."
  },
  "id": "mtsamples_replicate",
  "last_updated": "",
  "leaderboard_url": "https://crfm.stanford.edu/helm/medhelm/latest/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "mtsamples_procedures"
   ]
  },
  "measures": "This id is HELM's mtsamples_replicate scenario in MedHELM, not the surgical-only mtsamples_procedures wrap and not a third-party MTSamples leaderboard. Each item is an English transcribed report from MTSamples.com, stored as mtsamples_processed on raulista1997/benchmarkdata. HELM prefers a PLAN section as the reference, else SUMMARY, else FINDINGS, and removes only PLAN from the prompt. MedHELM places the task in clinical decision support / planning treatments. Display name on the schema is MTSamples.\n",
  "metric": {
   "baseline_note": "schema_medhelm.yaml display name is MTSamples Replicate Jury Score. HELM also attaches summarization metrics (BERTScore, distilbert-base-uncased). Default jury: GPT-4o, Llama 3.3 70B Instruct, Claude 3.7 Sonnet. LLMJuryMetric mean of 1-5 axis scores; default_score 1.0 if a judge fails. No published clinician baseline for this scenario. Not comparable to exact-match medical QA.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 5,
   "name": "mtsamples_replicate_accuracy (HELM LLM-jury average of accuracy, completeness, clarity, each 1-5)",
   "random_baseline": null,
   "unit": "points"
  },
  "name": "MTSamples Replicate (MedHELM)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2505.23802",
   "title": "Holistic evaluation of large language models for medical tasks with MedHELM",
   "url": "https://www.nature.com/articles/s41591-025-04151-2",
   "year": 2026
  },
  "publisher": {
   "authors": [
    "Suhana Bedi",
    "Hejie Cui",
    "Miguel Fuentes",
    "Alyssa Unell",
    "Michael Wornow",
    "Yifan Mai",
    "Percy Liang",
    "Nigam H. Shah"
   ],
   "org": "Stanford CRFM / MedHELM; source notes from MTSamples.com, packaged by raulista1997/benchmarkdata",
   "url": "https://crfm.stanford.edu/helm/medhelm/latest/"
  },
  "released": "2025",
  "repo_url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/mtsamples_replicate_scenario.py",
  "saturation": {
   "as_of": "",
   "note": "MedHELM overall win rates (DeepSeek R1 0.66, o3-mini 0.64 in the 2025 preprint) are suite-level, not this scenario. No mtsamples_replicate jury top score was copied from a rendered leaderboard.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM mtsamples_replicate_scenario.py (pin, PLAN-only strip, metadata)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/mtsamples_replicate_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM get_mtsamples_spec (replicate prompt, jury, BERTScore)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/run_specs/medhelm_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "schema_medhelm.yaml (MTSamples Replicate Jury Score; clinical decision support)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/static/schema_medhelm.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "MTSamplesReplicateAnnotator (treatment-plan 1-5 rubric)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/annotation/mtsamples_replicate_annotator.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM default jury models",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/medhelm/judges.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "raulista1997/benchmarkdata (1,269 processed .txt at pin)",
    "url": "https://github.com/raulista1997/benchmarkdata"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM arXiv 2505.23802",
    "url": "https://arxiv.org/abs/2505.23802"
   },
   {
    "accessed": "2026-09-08",
    "title": "Nature Medicine 2026 MedHELM article",
    "url": "https://www.nature.com/articles/s41591-025-04151-2"
   },
   {
    "accessed": "2026-09-08",
    "title": "MTSamples.com homepage (5,043 samples; educational-use credit request)",
    "url": "https://www.mtsamples.com/"
   }
  ],
  "status": "active",
  "subcategory": "treatment-plan generation from mixed-specialty transcribed clinical reports",
  "summary": "MedHELM wrap of mixed-specialty MTSamples notes: generate a treatment plan from a clinical transcription, scored by an LLM jury plus overlap metrics.",
  "tags": [
   "medical",
   "treatment-planning",
   "generation",
   "helm",
   "medhelm",
   "llm-jury"
  ],
  "task_format": "Zero-shot generation. HELM instructions: \"Given various information about a patient, return a reasonable treatment plan for the patient.\" No input noun. Output noun Answer. max_train_instances=0, max_tokens=512.\n"
 }
}