{
 "body": "\n## What it measures\n\nMedConfInfo asks whether an adolescent visit note contains sensitive material\nthat should be hidden from a parent or guardian who can see the portal. HELM's\nworked example is a checkup note in which a teenager discusses sexual activity\nand contraception in the private part of the visit. The model must answer A if\nthat kind of content is present and should be restricted, or B if not.\n\nThe task is English clinical text, single turn, closed labels. It is a privacy\nand confidentiality screen on real pediatric notes, not a general PHI de-identifier\nand not a chatbot safety eval.\n\n## How it is scored\n\nHELM uses joint multiple-choice prompting (\"Answer A or B.\") and exact match of\nthe letter. The methods appendix example gold label is B. Quasi-exact and prefix\nexact-match are logged beside the main exact_match metric. This page did not\nfind a published human-agreement number that applies to the 1000 MedHELM items.\nRabbani and colleagues assessed an LLM on confidential content in adolescent\nnotes in 2024, but their letter's numeric results were not independently\nre-read as full text here.\n\n## Dataset and licence\n\nExtended Data Table 1 reports 1000 instances evaluated, private and new. HELM\nloads medhelm-CONF-dataset_filtered.csv with prompt, context, and label, and\nmarks every instance as test. The notes are not on Hugging Face or GitHub.\n\nHELM cites Rabbani et al., JAMA Pediatrics, 2024\n(https://jamanetwork.com/journals/jamapediatrics/fullarticle/2814109). PubMed\nrecords the same title as DOI 10.1001/jamapediatrics.2023.6032, PMID 38252434,\n1 March 2024, pages 308-310. This page treats those as one article. The MedHELM\nCSV itself remains private; the research letter is not a substitute download.\n\nNo data licence was found. HELM code is Apache 2.0.\n\n## Who publishes it\n\nThe originating study authors on PubMed are Naveed Rabbani, Conner Brown,\nMichael Bedgood, Rachel L. Goldstein, Jennifer L. Carlson, Natalie M. Pageler,\nand Keith E. Morse, at Stanford Pediatrics and Lucile Packard Children's\nHospital. MedHELM (Bedi et al., Nature Medicine 2026; arXiv 2505.23802) wraps\nthe task as shc_conf_med. The HELM file landed in MedHELM V1 on 19 March 2025.\nCRFM hosts the leaderboard.\n\n## Lineage\n\nMedConfInfo is a Stanford pediatric confidentiality task, not a copy of\ni2b2/n2c2 de-identification shared tasks. It is also not PrivacyDetection\n(shc_privacy), which classifies portal messages rather than adolescent visit\nnotes. Sibling Stanford Health Care pages in this repository include shc_bmt, shc_cdi,\nshc_ent, shc_gip, shc_privacy, shc_proxy, shc_ptbm, shc_sei, and shc_sequoia\n(ClinicReferral). There is no MedHELM family page.\n\n## Saturation and contamination\n\nSaturation is unknown; no numeric MedConfInfo top score was parsed from the\nleaderboard. Contamination risk is low for the 1000 private items. The 2024\nletter is public and could leak example wording, but the authors still list\nMedConfInfo among datasets they will not redistribute.\n\n## How to run it\n\nUse HELM run spec `shc_conf_med` with `data_path` set to the private CSV.\nStanford's entries are in `run_entries_medhelm_private_stanford.conf`. Adapter\ninstructions are \"Answer A or B.\" The helper defaults to five in-context\nexamples, but this scenario has no train split, so the run is effectively\nzero-shot. Outside groups cannot rebuild the official score without the file.\nNo lm-eval, Inspect, OpenCompass, or BIG-bench twin was confirmed.\n\n## Reading the numbers\n\nA high exact-match score means the model usually matched the gold A/B label on\nwhether these adolescent notes should be restricted from a parent view. It does\nnot certify a portal privacy filter: the items are from one institution, the\nlabels are hidden, and exact match does not score which span is confidential.\nRead it next to PrivacyDetection and ProxySender, which cover related pediatric\ncommunication risks with different inputs.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "shc_conf",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "shc_conf_med",
   "MedConfInfo"
  ],
  "category": "domain",
  "contamination": {
   "note": "The 1000 MedHELM items are listed as a private dataset that is not redistributed. Nature Medicine groups 14 such private sets, not all from Stanford Health Care. The 2024 JAMA Pediatrics research letter is public, but that letter is not the HELM CSV.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": 1000,
   "size_note": "Nature Medicine Extended Data Table 1 reports 1000 instances evaluated, private and new. HELM reads medhelm-CONF-dataset_filtered.csv with prompt, context, and label columns and tags every row as test.\n",
   "splits": "test only (HELM TEST_SPLIT); no published train or validation split",
   "url": ""
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-001 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-001"
  },
  "harness": {
   "bigbench": "",
   "helm": "shc_conf_med",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "shc_conf",
  "last_updated": "2025-08",
  "leaderboard_url": "https://crfm.stanford.edu/helm/medhelm/latest/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "shc_bmt",
    "shc_cdi",
    "shc_ent",
    "shc_gip",
    "shc_privacy",
    "shc_proxy",
    "shc_ptbm",
    "shc_sei",
    "shc_sequoia"
   ]
  },
  "measures": "MedConfInfo tests whether a model can read an English adolescent encounter note and decide if it contains sensitive protected health information that should be restricted from parental access under adolescent confidentiality rules. HELM's example mentions sexual activity, mental health, or substance use discussed in a private portion of a visit. The census id is shc_conf; the runnable HELM scenario is shc_conf_med.\n",
  "metric": {
   "baseline_note": "HELM scores exact match of A or B. Rabbani et al. 2024 studied LLM classification of confidential content in adolescent notes, but this page did not recover a numeric human baseline that applies to the 1000-item MedHELM split.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1.0,
   "name": "exact_match",
   "random_baseline": null,
   "unit": "0-1 scale"
  },
  "name": "MedConfInfo",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "Evaluation of a Large Language Model to Identify Confidential Content in Adolescent Encounter Notes",
   "url": "https://pubmed.ncbi.nlm.nih.gov/38252434/",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Naveed Rabbani",
    "Conner Brown",
    "Michael Bedgood",
    "Rachel L. Goldstein",
    "Jennifer L. Carlson",
    "Natalie M. Pageler",
    "Keith E. Morse"
   ],
   "org": "Stanford University Department of Pediatrics, Stanford Health Care, and Stanford CRFM (MedHELM)",
   "url": "https://pubmed.ncbi.nlm.nih.gov/38252434/"
  },
  "released": "2025-03",
  "repo_url": "https://github.com/stanford-crfm/helm",
  "saturation": {
   "as_of": "",
   "note": "No dated numeric top exact-match for MedConfInfo was recovered from a parsed leaderboard table during this research.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM shc_conf_scenario.py (SHCCONFMedScenario, MedConfInfo)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/shc_conf_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM medhelm_run_specs.py (shc_conf_med run spec)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/run_specs/medhelm_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM run_entries_medhelm_private_stanford.conf",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/presentation/run_entries_medhelm_private_stanford.conf"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM schema_medhelm.yaml (MedConfInfo group)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/static/schema_medhelm.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "Rabbani et al., JAMA Pediatrics 2024 (PMID 38252434)",
    "url": "https://pubmed.ncbi.nlm.nih.gov/38252434/"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM arXiv:2505.23802",
    "url": "https://arxiv.org/abs/2505.23802"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM Nature Medicine author manuscript (PMC13267972)",
    "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC13267972/"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM documentation",
    "url": "https://crfm-helm.readthedocs.io/en/latest/medhelm/"
   },
   {
    "accessed": "2026-09-08",
    "title": "stanford-crfm/helm Apache License 2.0",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/LICENSE"
   }
  ],
  "status": "active",
  "subcategory": "adolescent note confidentiality for parental access",
  "summary": "Private MedHELM task from Stanford adolescent notes: say whether a visit note contains sensitive content that should be withheld from a parent portal view.\n",
  "tags": [
   "medical",
   "clinical",
   "classification",
   "privacy",
   "pediatrics",
   "private-dataset",
   "medhelm",
   "stanford-health-care"
  ],
  "task_format": "Joint multiple-choice generation. HELM asks for A if the note contains sensitive content that should be restricted, or B if not, with no extra text.\n"
 }
}