{
 "body": "\n## What it measures\n\nENT-Referral asks whether a patient's note supports sending that patient to an\near, nose, and throat specialist. HELM's methods example is a 52-year-old with\nrecurrent sinus infections and nasal complaints. The model must choose A if a\nreferral is supported, B if it is not, or C if the note never mentions the\nissue. That third option is the main protocol difference from binary Stanford\nHealth Care tasks such as CDI-QA or HospiceReferral.\n\nThe input is an English consultation note. The skill is referral triage from\nunstructured text, not otolaryngology board-style knowledge.\n\n## How it is scored\n\nThe run spec instructs the model to \"Answer A, B, or C.\" HELM scores exact\nmatch of that letter. The paper describes the metric as the fraction of\ninstances whose classification matches the reference, including the\nno-mention class. No published class mix was found, so a 33 percent chance\nline is not established. Quasi-exact and prefix exact-match are recorded\nalongside exact_match.\n\n## Dataset and licence\n\nExtended Data Table 1 reports 1000 instances evaluated, private and new. HELM\nloads medhelm-ENT-dataset_filtered.csv. The scenario skips rows with an empty\nlabel and comments \"skip rows with character/encoding issues - 79\". Whether 79\nis a skipped-row count, a ticket id, or something else was not established;\nthe paper's 1000 is the figure this page uses for size.\n\nThe notes are Stanford Health Care EHR text and are not released. No data\nlicence was found. HELM code is Apache 2.0.\n\n## Who publishes it\n\nENT-Referral is a new MedHELM administration benchmark from the Stanford\nHealth Care and CRFM collaboration (Bedi, Cui, Fuentes, Unell, Shah and\ncolleagues). The preprint is arXiv:2505.23802 (May 2025); the journal article\nis Nature Medicine, 20 January 2026. The HELM scenario was added with MedHELM\nV1 on 19 March 2025. Taxonomy: Administration and Workflow, care coordination\nand planning.\n\n## Lineage\n\nThis is not a public ENT exam set and not ClinicReferral (shc_sequoia_med),\nwhich targets the Sequoia clinic from palliative notes. It is also not\nHospiceReferral (shc_gip), which is a binary hospice-eligibility task. Sibling pages include shc_bmt, shc_cdi, shc_conf, shc_gip, shc_privacy,\nshc_proxy, shc_ptbm, shc_sei, and shc_sequoia (ClinicReferral). There is no\nMedHELM family page.\n\n## Saturation and contamination\n\nSaturation is unknown without a parsed per-task top score. Contamination risk\nis low because the CSV is private. The three-way label set also makes\naccidental copying from binary yes/no medical quizzes less likely to transfer\ncleanly.\n\n## How to run it\n\nRun HELM scenario `shc_ent_med` with the private `data_path`. Stanford entries\nare in `run_entries_medhelm_private_stanford.conf`. Adapter instructions are\n\"Answer A, B, or C.\" HELM also prepends a 1-based counter to each prompt, so a\nreimplementation that drops that counter is not the same task. The adapter\ndefaults to five shots, but only a test split exists, so the official protocol\nis effectively zero-shot. No other harness was confirmed.\n\n## Reading the numbers\n\nA high score means the model usually picked the gold yes, no, or no-mention\nlabel on these ENT triage notes. It does not mean the model should auto-place\nreferrals: C (no mention) is a valid gold class, and a model that never uses C\ncan look strong on a two-class subset while failing the published protocol.\nCompare only with other MedHELM exact-match referral tasks, and check that the\nrun allowed three letters.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "shc_ent",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "shc_ent_med",
   "ENT-Referral"
  ],
  "category": "domain",
  "contamination": {
   "note": "Listed among the 14 private MedHELM datasets that are not redistributed, which limits verbatim training leakage of this item set. Those 14 are not all Stanford Health Care records.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": 1000,
   "size_note": "Nature Medicine Extended Data Table 1 reports 1000 instances evaluated. HELM reads medhelm-ENT-dataset_filtered.csv and skips rows whose label cell is empty, with a code comment \"skip rows with character/encoding issues - 79\" whose exact meaning (skipped-row count versus an issue id) is not established.\n",
   "splits": "test only (HELM TEST_SPLIT); empty-label rows dropped; no train split",
   "url": ""
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-001 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-001"
  },
  "harness": {
   "bigbench": "",
   "helm": "shc_ent_med",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "shc_ent",
  "last_updated": "2025-08",
  "leaderboard_url": "https://crfm.stanford.edu/helm/medhelm/latest/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "shc_bmt",
    "shc_cdi",
    "shc_conf",
    "shc_gip",
    "shc_privacy",
    "shc_proxy",
    "shc_ptbm",
    "shc_sei",
    "shc_sequoia"
   ]
  },
  "measures": "ENT-Referral tests whether a model can read an unstructured English clinical note and decide if the note supports referring the patient to an ear, nose, and throat (ENT) specialist. Unlike the binary Stanford Health Care tasks, HELM allows a third label, C, for no mention of referral. The census id is shc_ent; the runnable HELM scenario is shc_ent_med.\n",
  "metric": {
   "baseline_note": "Main metric is exact_match among A, B, and C. A one-third chance baseline would require equal class sizes, which were not published.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1.0,
   "name": "exact_match",
   "random_baseline": null,
   "unit": "0-1 scale"
  },
  "name": "ENT-Referral",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2505.23802",
   "title": "Holistic evaluation of large language models for medical tasks with MedHELM",
   "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC13267972/",
   "year": 2026
  },
  "publisher": {
   "authors": [
    "Suhana Bedi",
    "Hejie Cui",
    "Miguel Fuentes",
    "Alyssa Unell",
    "Nigam H. Shah"
   ],
   "org": "Stanford Health Care and Stanford CRFM (MedHELM)",
   "url": "https://crfm.stanford.edu/helm/medhelm/latest/"
  },
  "released": "2025-03",
  "repo_url": "https://github.com/stanford-crfm/helm",
  "saturation": {
   "as_of": "",
   "note": "No dated numeric top exact-match for ENT-Referral was recovered from a parsed leaderboard table.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM shc_ent_scenario.py (SHCENTMedScenario, ENT-Referral)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/shc_ent_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM medhelm_run_specs.py (shc_ent_med run spec)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/run_specs/medhelm_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM run_entries_medhelm_private_stanford.conf",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/presentation/run_entries_medhelm_private_stanford.conf"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM schema_medhelm.yaml (ENT-Referral group)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/static/schema_medhelm.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM arXiv:2505.23802",
    "url": "https://arxiv.org/abs/2505.23802"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM Nature Medicine author manuscript (PMC13267972)",
    "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC13267972/"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM documentation",
    "url": "https://crfm-helm.readthedocs.io/en/latest/medhelm/"
   },
   {
    "accessed": "2026-09-08",
    "title": "stanford-crfm/helm Apache License 2.0",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/LICENSE"
   }
  ],
  "status": "active",
  "subcategory": "ear, nose, and throat specialist referral from notes",
  "summary": "Private Stanford Health Care MedHELM task: from a clinical note, answer whether an ENT referral is supported, with A yes, B no, or C no mention.\n",
  "tags": [
   "medical",
   "clinical",
   "classification",
   "referral",
   "private-dataset",
   "medhelm",
   "stanford-health-care"
  ],
  "task_format": "Joint multiple-choice generation. HELM prefixes a running item counter, then asks for A (yes), B (no), or C (no mention), with no extra text.\n"
 }
}