{
 "body": "\n## What it measures\n\nPrivacyDetection asks whether a patient-portal message leaks confidential\ninformation. HELM's methods example is a short portal note (\"Hi, just wanted to\nconfirm that my test came\u2026\") and a yes/no question about confidential content.\nThe required answer is A for yes or B for no.\n\nTwo HELM texts disagree about how the messages were made. The class docstring\nsays the messages were generated by an LLM from patient notes, then checked for\nconfidential content. The published metadata and Nature Medicine table describe\nportal messages from patients or caregivers. This page records both readings and\ndoes not pick one.\n\n## How it is scored\n\nHELM uses joint multiple-choice adaptation with \"Answer A or B.\" and exact\nmatch of the letter. The methods appendix also shows a second output format\n(\"privacy-leaking or non-privacy-leaking\") in the same PrivacyDetection write-up;\nthe runnable scenario file still demands A or B. Official MedHELM numbers should\nbe read as letter exact match unless a report says otherwise. No class prior or\nhuman baseline for the 300 items was found.\n\n## Dataset and licence\n\nExtended Data Table 1 reports 300 instances evaluated, private and new, grouped\nwith ProxySender as citation ME38. HELM loads\nmedhelm-PRIVACY-dataset_filtered.csv. All instances are test. The CSV is not\npublic. No data licence was found. HELM code is Apache 2.0.\n\n## Who publishes it\n\nHELM and MedHELM cite Tse, Zahedivash, Anoshiravani, Carlson, Haberkorn, and\nMorse, JAMA Pediatrics, 1 January 2025, DOI 10.1001/jamapediatrics.2024.4438,\nPMID 39495530, pages 93-94. PubMed's plain-language summary of that letter is\nabout telling adolescent patient messages from parent or guardian messages and\nthen drafting a reply. That is closer to ProxySender than to a leak classifier.\nTreat Tse et al. as the cited pediatric-portal paper, not as proof that\nPrivacyDetection is the letter's published generation task.\n\nMedHELM (Bedi et al., 2026) hosts the harness. The HELM file was added on\n8 April 2025 in the \"SHC additional datasets\" change, after MedHELM V1.\n\n## Lineage\n\nThis is not MedConfInfo (shc_conf), which labels adolescent visit notes for\nparental access. It is not ProxySender (shc_proxy), which labels who sent the\nmessage. Both privacy and proxy cite the same 2025 letter and share the ME38\nfootnote; they remain separate HELM scenarios. Other Stanford Health Care siblings include shc_bmt, shc_cdi, shc_ent, shc_gip,\nshc_ptbm, shc_sei, and shc_sequoia. There is no MedHELM family page.\n\n## Saturation and contamination\n\nSaturation is unknown. Contamination risk is low for the private 300 items. The\ncited letter is public and short; it could leak task framing without leaking\nthis CSV.\n\n## How to run it\n\nRun HELM scenario `shc_privacy_med` with the private `data_path`. Stanford\nentries are in `run_entries_medhelm_private_stanford.conf`. Adapter\ninstructions are \"Answer A or B.\" The helper defaults to five shots, but there\nis no train split, so the run is effectively zero-shot. Do not compare a\nfree-text \"privacy-leaking\" label run with the official A/B exact-match. No\nother harness was confirmed.\n\n## Reading the numbers\n\nA high score means the model usually matched the gold yes or no on whether\nthese 300 messages were judged confidential. It is not a HIPAA certification\nand not a jailbreak eval. Because HELM's own description of the source messages\nconflicts, check which construction a report used before treating two\nPrivacyDetection numbers as the same task. Read it with ProxySender and\nMedConfInfo, not with general safety averages.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "shc_privacy",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "shc_privacy_med",
   "PrivacyDetection"
  ],
  "category": "domain",
  "contamination": {
   "note": "The 300 items are listed as a private dataset. Nature Medicine groups 14 such private sets, not all from Stanford Health Care. HELM cites Tse et al. 2025, a public research letter whose full item set is not this CSV.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": 300,
   "size_note": "Nature Medicine Extended Data Table 1 reports 300 instances evaluated, private and new, cited as ME38 with ProxySender. HELM reads medhelm-PRIVACY-dataset_filtered.csv with prompt, context, and label.\n",
   "splits": "test only (HELM TEST_SPLIT); no published train or validation split",
   "url": ""
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-001 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-001"
  },
  "harness": {
   "bigbench": "",
   "helm": "shc_privacy_med",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "shc_privacy",
  "last_updated": "2025-08",
  "leaderboard_url": "https://crfm.stanford.edu/helm/medhelm/latest/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "shc_bmt",
    "shc_cdi",
    "shc_conf",
    "shc_ent",
    "shc_gip",
    "shc_proxy",
    "shc_ptbm",
    "shc_sei",
    "shc_sequoia"
   ]
  },
  "measures": "PrivacyDetection tests whether a model can read an English patient-portal message and say whether it contains confidential or privacy-leaking information that should be protected. HELM metadata describes messages from patients or caregivers. The census id is shc_privacy; the runnable HELM scenario is shc_privacy_med.\n",
  "metric": {
   "baseline_note": "Main metric is exact_match of A or B. No published class balance or human privacy-officer baseline for the 300 MedHELM items was found.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1.0,
   "name": "exact_match",
   "random_baseline": null,
   "unit": "0-1 scale"
  },
  "name": "PrivacyDetection",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "Large Language Model Responses to Adolescent Patient and Proxy Messages",
   "url": "https://pubmed.ncbi.nlm.nih.gov/39495530/",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Gabriel Tse",
    "Aydin Zahedivash",
    "Arash Anoshiravani",
    "Jennifer Carlson",
    "William Haberkorn",
    "Keith E. Morse"
   ],
   "org": "Stanford University Department of Pediatrics, Stanford Health Care, and Stanford CRFM (MedHELM)",
   "url": "https://pubmed.ncbi.nlm.nih.gov/39495530/"
  },
  "released": "2025-04",
  "repo_url": "https://github.com/stanford-crfm/helm",
  "saturation": {
   "as_of": "",
   "note": "No dated numeric top exact-match for PrivacyDetection was recovered from a parsed leaderboard table.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM shc_privacy_scenario.py (SHCPRIVACYMedScenario, PrivacyDetection)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/shc_privacy_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM medhelm_run_specs.py (shc_privacy_med run spec)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/run_specs/medhelm_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM run_entries_medhelm_private_stanford.conf",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/presentation/run_entries_medhelm_private_stanford.conf"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM schema_medhelm.yaml (PrivacyDetection group)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/static/schema_medhelm.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "Tse et al., JAMA Pediatrics 2025 (PMID 39495530)",
    "url": "https://pubmed.ncbi.nlm.nih.gov/39495530/"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM arXiv:2505.23802",
    "url": "https://arxiv.org/abs/2505.23802"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM Nature Medicine author manuscript (PMC13267972)",
    "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC13267972/"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM documentation",
    "url": "https://crfm-helm.readthedocs.io/en/latest/medhelm/"
   },
   {
    "accessed": "2026-09-08",
    "title": "stanford-crfm/helm Apache License 2.0",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/LICENSE"
   }
  ],
  "status": "active",
  "subcategory": "confidential content in patient portal messages",
  "summary": "Private MedHELM task: decide whether a patient-portal message contains confidential or privacy-leaking information, answering A for yes or B for no.\n",
  "tags": [
   "medical",
   "clinical",
   "classification",
   "privacy",
   "patient-portal",
   "private-dataset",
   "medhelm",
   "stanford-health-care"
  ],
  "task_format": "Joint multiple-choice generation. HELM instructs the model to review clinical messages for confidential information and answer A for yes or B for no.\n"
 }
}