{
 "body": "\n## What it measures\n\nProxySender asks who wrote a message that landed in a clinician's inbox: the\npatient, or a proxy such as a parent or spouse. HELM's example is a portal\nmessage that identifies the sender as \"Jane's mom.\" The published MedHELM\nprompt asks whether the message was sent by a proxy user, with A for yes and B\nfor no. Language is English. Modality is text.\n\nThis is a sender-identity screen, not a clinical-advice generator. PubMed's\nsummary of the cited letter also mentions drafting an appropriate reply; HELM's\nscenario does not score that generation step.\n\n## How it is scored\n\nHELM scores exact match of the output letter to the gold label. The Nature\nMedicine methods text still shows a binary A/B prompt and a binary gold\nexample. On 16 October 2025, HELM pull request 3911 changed\nPOSSIBLE_ANSWER_CHOICES from [A, B] to [A, B, C] and changed adapter\ninstructions to \"Answer A, B, or C.\" The instance prompt in\n`create_benchmark` still tells the model to answer A for yes or B for no and\nnever defines C. A report should say which code revision it used. No class\nprior or human baseline for the 300 items was published.\n\n## Dataset and licence\n\nExtended Data Table 1 reports 300 instances evaluated, private and new, cited\nwith PrivacyDetection as ME38. HELM loads medhelm-PROXY-dataset_filtered.csv\nwith prompt, context, and label, and tags every row as test. The messages are\nStanford Health Care portal text and are not released. No data licence was\nfound. HELM code is Apache 2.0.\n\n## Who publishes it\n\nThe cited letter is Tse, Zahedivash, Anoshiravani, Carlson, Haberkorn, and\nMorse, JAMA Pediatrics, 1 January 2025, DOI 10.1001/jamapediatrics.2024.4438,\nPMID 39495530, pages 93-94. HELM added the scenario on 8 April 2025. MedHELM\n(Bedi et al., Nature Medicine 2026; arXiv 2505.23802) reports the suite score.\nTaxonomy: Patient Communication and Education, patient-provider messaging.\n\n## Lineage\n\nProxySender is not PrivacyDetection, which asks whether a message leaks\nconfidential facts, and not MedConfInfo, which screens adolescent visit notes\nfor parental access. Those three tasks are related pediatric communication\nchecks, not one evaluation under three names. Other Stanford Health Care siblings include shc_bmt, shc_cdi, shc_ent, shc_gip,\nshc_ptbm, shc_sei, and shc_sequoia. There is no MedHELM family page.\n\n## Saturation and contamination\n\nSaturation is unknown. The October 2025 third-label change is a further reason\nnot to treat early and late HELM runs as one series. Contamination risk is low\nfor the private CSV. The public letter could leak the idea of the task without\nleaking these 300 messages.\n\n## How to run it\n\nRun HELM scenario `shc_proxy_med` with the private `data_path`. Stanford\nentries are in `run_entries_medhelm_private_stanford.conf`. Current adapter\ninstructions are \"Answer A, B, or C.\" The instance string still says A or B.\nThe multiple-choice helper defaults to five shots, but only a test split\nexists, so the run is effectively zero-shot. Pin the HELM revision when\ncomparing scores. No lm-eval, Inspect, OpenCompass, or BIG-bench twin was\nconfirmed.\n\n## Reading the numbers\n\nA high exact-match score means the model usually matched the gold sender label\non these 300 portal messages under HELM's letter protocol. It does not mean the\nmodel wrote a safe reply, which the cited letter also discusses and HELM does\nnot score here. Check whether C was a legal label in that run, and read the\nnumber next to PrivacyDetection rather than as a general pediatric-chat score.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "shc_proxy",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "shc_proxy_med",
   "ProxySender"
  ],
  "category": "domain",
  "contamination": {
   "note": "The 300 items are a private dataset. Nature Medicine groups 14 such private sets, not all from Stanford Health Care. The cited 2025 research letter is public but is not a download of this CSV.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": 300,
   "size_note": "Nature Medicine Extended Data Table 1 reports 300 instances evaluated, private and new, cited as ME38 with PrivacyDetection. HELM reads medhelm-PROXY-dataset_filtered.csv with prompt, context, and label.\n",
   "splits": "test only (HELM TEST_SPLIT); no published train or validation split",
   "url": ""
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-001 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-001"
  },
  "harness": {
   "bigbench": "",
   "helm": "shc_proxy_med",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "shc_proxy",
  "last_updated": "2025-10",
  "leaderboard_url": "https://crfm.stanford.edu/helm/medhelm/latest/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "shc_bmt",
    "shc_cdi",
    "shc_conf",
    "shc_ent",
    "shc_gip",
    "shc_privacy",
    "shc_ptbm",
    "shc_sei",
    "shc_sequoia"
   ]
  },
  "measures": "ProxySender tests whether a model can read an English patient-portal message sent to a clinician and decide if a proxy user (parent, spouse, or other caregiver) sent it rather than the patient. HELM's example is a message that begins \"Hi Dr. Lee, this is Jane's mom.\" The census id is shc_proxy; the runnable HELM scenario is shc_proxy_med.\n",
  "metric": {
   "baseline_note": "Main metric is exact_match. Gold labels are asserted to be in {A, B, C} after the October 2025 fix. Class mix and a human baseline for the 300 items were not published.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1.0,
   "name": "exact_match",
   "random_baseline": null,
   "unit": "0-1 scale"
  },
  "name": "ProxySender",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "Large Language Model Responses to Adolescent Patient and Proxy Messages",
   "url": "https://pubmed.ncbi.nlm.nih.gov/39495530/",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Gabriel Tse",
    "Aydin Zahedivash",
    "Arash Anoshiravani",
    "Jennifer Carlson",
    "William Haberkorn",
    "Keith E. Morse"
   ],
   "org": "Stanford University Department of Pediatrics, Stanford Health Care, and Stanford CRFM (MedHELM)",
   "url": "https://pubmed.ncbi.nlm.nih.gov/39495530/"
  },
  "released": "2025-04",
  "repo_url": "https://github.com/stanford-crfm/helm",
  "saturation": {
   "as_of": "",
   "note": "No dated numeric top exact-match for ProxySender was recovered from a parsed leaderboard table. The October 2025 label-set change can make older and newer runs hard to compare.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM shc_proxy_scenario.py (SHCPROXYMedScenario, ProxySender)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/shc_proxy_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM commit Fix Proxy scenario (#3911), 2025-10-16",
    "url": "https://github.com/stanford-crfm/helm/commit/a363f41c"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM medhelm_run_specs.py (shc_proxy_med run spec)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/run_specs/medhelm_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM run_entries_medhelm_private_stanford.conf",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/presentation/run_entries_medhelm_private_stanford.conf"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM schema_medhelm.yaml (ProxySender group)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/static/schema_medhelm.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "Tse et al., JAMA Pediatrics 2025 (PMID 39495530)",
    "url": "https://pubmed.ncbi.nlm.nih.gov/39495530/"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM arXiv:2505.23802",
    "url": "https://arxiv.org/abs/2505.23802"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM Nature Medicine author manuscript (PMC13267972)",
    "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC13267972/"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM documentation",
    "url": "https://crfm-helm.readthedocs.io/en/latest/medhelm/"
   },
   {
    "accessed": "2026-09-08",
    "title": "stanford-crfm/helm Apache License 2.0",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/LICENSE"
   }
  ],
  "status": "active",
  "subcategory": "patient versus proxy sender of portal messages",
  "summary": "Private MedHELM task: decide whether a clinician-facing portal message was sent by the patient or by a proxy such as a parent or spouse.\n",
  "tags": [
   "medical",
   "clinical",
   "classification",
   "patient-portal",
   "pediatrics",
   "private-dataset",
   "medhelm",
   "stanford-health-care"
  ],
  "task_format": "Joint multiple-choice generation. The instance prompt still says A for yes (proxy) or B for no. After 16 October 2025 the adapter also names C as a legal letter, while the prompt text was not updated to define C.\n"
 }
}