{
 "body": "\n## What it measures\n\nClinicReferral asks a model to answer yes/no questions about whether a patient should be referred to Stanford's Sequoia clinic, using text from palliative-care notes. HELM feeds a numbered question and a context field and requires `A` or `B` only. The MedHELM schema describes the skill as pre-referral classification for a hospital administrator. Unlike the two ADHD SHC tasks, the scenario file does not name an age group, a year range, or a journal paper. The only primary descriptions are the HELM docstring (\"manually curated answers to several questions\") and the MedHELM table row (\"manually curated answers to questions regarding patient referrals to the Sequoia clinic\").\n\nThe task is workflow classification on private notes, not a public referral-policy exam. What \"Sequoia clinic\" covers beyond that HELM sentence is not established from the sources opened for this page.\n\n## How it is scored\n\nHELM uses exact match on the A/B label, test split, with multiple-choice joint adaptation and the instruction \"Answer A or B.\" Exact match is a 0\u20131 fraction; this page writes random guessing as 50% on two options. The helper defaults to five in-context examples, but the scenario tags every CSV row as `TEST_SPLIT`, so the protocol is effectively zero-shot. No published human baseline, majority baseline, or model exact-match for this CSV was found. The MedHELM paper's 0.050 \u00b1 0.011 figure for ClinicReferral is a minimum detectable effect in Appendix E, not a score.\n\n## Dataset and licence\n\nThe scenario reads a local CSV with `question`, `context`, and `label` (not the `prompt`/`context`/`label` columns used by the ADHD scenarios). Private Stanford run entries name `medhelm-sequoia-dataset_filtered.csv`. Nature Medicine Table 1 reports 326 instances evaluated. Class balance and a data licence are not published. MedHELM marks the benchmark private and lists it among fourteen Stanford Health Care datasets blocked by institutional agreements. There is no public dataset URL.\n\n## Who publishes it\n\nStanford CRFM publishes the evaluation as part of MedHELM (Bedi, Cui, Fuentes, Unell, Wornow, and co-authors). The peer-reviewed suite paper is Nature Medicine, 20 January 2026 (doi:10.1038/s41591-025-04151-2; PMC13267972; arXiv:2505.23802, 26 May 2025). The notes come from Stanford Health Care. No separate Sequoia-clinic methods paper was cited in the scenario. The MedHELM author list is long; this page does not copy it in full. Display name ClinicReferral and scenario name `shc_sequoia_med` are from HELM's schema and Python.\n\n## Lineage\n\nThis page is the HELM scenario whose census id is `shc_sequoia`. It sits with other private SHC MedHELM tasks (`shc_cdi`, `shc_ent`, `shc_gip`) under Administration and Workflow. It is not the ADHD pair `shc_ptbm` / `shc_sei`, which use different notes and questions. This repository has no MedHELM family page. No predecessor dataset was named.\n\n## Saturation and contamination\n\nSaturation is unknown: no public exact-match table was recovered from the JavaScript leaderboard. Contamination risk is low if the notes never left SHC, which is the access model MedHELM documents for private benchmarks. That is an access claim, not a measured membership test.\n\n## How to run it\n\nRun HELM spec `shc_sequoia_med` with `data_path` pointing at the private CSV. Official reproductions use `run_entries_medhelm_private_stanford.conf` and Stanford Health Care model deployments. Adapter instructions are \"Answer A or B.\" The multiple-choice helper defaults to five shots, but only test instances exist, so official rows are effectively zero-shot. Public MedHELM install docs (`docs/medhelm.md`) explain public vs gated vs private run-entry files; this scenario is private. No lm-evaluation-harness, inspect_evals, or OpenCompass task was found.\n\n## Reading the numbers\n\nA high ClinicReferral score means the model matches SHC's curated A/B labels on this unpublished referral CSV under HELM's prompt. It does not show that the model can run a palliative-care clinic, apply another health system's referral rules, or answer free-text referral questions. Nature Medicine evaluated 326 instances; class balance and current leaderboard cells are still not public as static text. Treat any leaked number as incomparable unless the reporter names this HELM spec and this CSV. If you need a public administration-style clinical task, this is the wrong row; if you need ADHD note classification, use `shc_ptbm` or `shc_sei`.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "shc_sequoia",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "shc_sequoia_med",
   "Sequoia clinic referral"
  ],
  "category": "domain",
  "contamination": {
   "note": "MedHELM marks the benchmark private. HELM run entries appear only in run_entries_medhelm_private_stanford.conf, with an internal Databricks CSV path. No public item dump was found, so public pretraining is unlikely to include the notes or labels.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": 326,
   "size_note": "Nature Medicine Table 1 (PMC13267972) reports 326 instances evaluated for ClinicReferral, private and new. HELM still loads a private CSV (medhelm-sequoia-dataset_filtered.csv) with columns question, context, and label; this page does not independently count that file. Unlike shc_ptbm and shc_sei, no separate methods paper describing this item set was cited from the scenario.\n",
   "splits": "HELM exposes a single test split from a local CSV; no public train/validation/test counts",
   "url": ""
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-002 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-002"
  },
  "harness": {
   "bigbench": "",
   "helm": "shc_sequoia_med",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "MedHELM private Stanford run entries; helm-run needs data_path to a local CSV"
  },
  "id": "shc_sequoia",
  "last_updated": "2026-01",
  "leaderboard_url": "https://crfm.stanford.edu/helm/medhelm/latest",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "ClinicReferral (HELM scenario shc_sequoia_med) tests whether a model can read English palliative-care notes and answer curated yes/no questions about referral to the Sequoia clinic. HELM's scenario docstring calls the items \"manually curated answers to several questions for Sequoia clinic referrals.\" The prompt numbers each item and requires `A` (yes) or `B` (no) with no extra text. MedHELM places the task in Administration and Workflow and describes it as determining referral eligibility from palliative-care notes to help automate clinic workflows.\n",
  "metric": {
   "baseline_note": "Two-way A/B choice gives a 50% random-guess rate. No human accuracy, inter-annotator agreement, or published model exact-match on this HELM CSV was found in the scenario file, schema, or MedHELM paper.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "exact_match",
   "random_baseline": 50,
   "unit": "%"
  },
  "name": "ClinicReferral",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2505.23802",
   "title": "Holistic evaluation of large language models for medical tasks with MedHELM",
   "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC13267972/",
   "year": 2026
  },
  "publisher": {
   "authors": [],
   "org": "Stanford Health Care / Stanford CRFM (MedHELM)",
   "url": "https://crfm.stanford.edu/helm/medhelm/latest"
  },
  "released": "2025-05",
  "repo_url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/shc_sequoia_scenario.py",
  "saturation": {
   "as_of": "",
   "note": "MedHELM lists ClinicReferral as a private, newly formulated benchmark. No static exact-match top score was read from the public leaderboard page. Table 8 of the MedHELM paper reports a minimum detectable effect of 0.050 \u00b1 0.011 for ClinicReferral, which is a sample-size statistic, not an accuracy.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM shc_sequoia_scenario.py (SHCSequoiaMedScenario)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/shc_sequoia_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM medhelm_run_specs.py (shc_sequoia_med run spec)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/run_specs/medhelm_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM schema_medhelm.yaml (ClinicReferral)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/static/schema_medhelm.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM documentation (access levels)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/docs/medhelm.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM private Stanford run entries (shc_sequoia_med CSV path)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/presentation/run_entries_medhelm_private_stanford.conf"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks (arXiv:2505.23802)",
    "url": "https://arxiv.org/abs/2505.23802"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM full text (ar5iv)",
    "url": "https://ar5iv.labs.arxiv.org/html/2505.23802"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM leaderboard (JavaScript shell; scores not recovered as static text)",
    "url": "https://crfm.stanford.edu/helm/medhelm/latest"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM Nature Medicine author manuscript (PMC13267972), Table 1 instances evaluated",
    "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC13267972/"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM get_multiple_choice_adapter_spec default max_train_instances=5",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/adaptation/common_adapter_specs.py"
   }
  ],
  "status": "active",
  "subcategory": "palliative-care note classification: Sequoia clinic referral eligibility",
  "summary": "Private MedHELM binary task: from a palliative-care note, decide whether a patient is eligible for referral to Stanford's Sequoia clinic.",
  "tags": [
   "biomedical",
   "clinical-notes",
   "referral",
   "palliative-care",
   "medhelm",
   "private",
   "classification"
  ],
  "task_format": "Binary A/B classification over a question plus note context; English text; HELM multiple-choice joint adaptation."
 }
}