{
 "body": "\n## What it measures\n\nCDI-QA is a chart-verification task, not a billing-code extractor and not a\ndiagnosis exam. The model is given an English hospital admission note and a\nquestion that a clinical documentation integrity review would ask, for example\nwhether the note supports a named condition. HELM's methods example uses\nfindings relevant to verifying a condition such as acute kidney injury. The\nrequired output is A for yes or B for no.\n\nThe work is hospital operations: queries written against prior notes so that a\nmodel, or a human CDI specialist, can confirm that the chart backs a clinical\nstatement. Language is English. Modality is text.\n\n## How it is scored\n\nHELM uses joint multiple-choice adaptation and exact match of the single letter\nto the gold label. The schema lists exact_match on test as the main metric;\nquasi-exact and prefix variants are also computed. The paper's gold-standard\nexample for this task is the letter A. No human CDI-reviewer baseline or class\nprior was published, so a 50 percent chance line is not established.\n\n## Dataset and licence\n\nExtended Data Table 1 reports 1000 instances evaluated, private and newly\ncurated. HELM reads prompt, context, and label from a private CSV named\nmedhelm-CDI-dataset_filtered.csv in the Stanford run file. All rows become test\ninstances. The HELM class docstring and Nature Medicine Extended Data Table 1 say\n\"Clinical Document Integrity.\" The methods appendix and HELM metadata say\n\"Clinical Documentation Integrity.\" This page records both spellings.\n\nThe notes are not public. Nature Medicine lists CDI-QA among 14 private\ndatasets blocked by institutional agreements and patient privacy; that group\nis not limited to Stanford Health Care. No data licence was found. HELM code\nis Apache 2.0.\n\n## Who publishes it\n\nCDI-QA is a MedHELM private benchmark from Stanford Health Care's partnership\nwith Stanford CRFM. The peer-reviewed suite paper is Bedi, Cui, Fuentes, Unell\nand colleagues, Nature Medicine, 20 January 2026 (arXiv:2505.23802, May 2025).\nThe journal lists Bedi as corresponding author.\nThe HELM scenario shipped in MedHELM V1 on 19 March 2025. The live board is\nhttps://crfm.stanford.edu/helm/medhelm/latest/.\n\n## Lineage\n\nThis is a new EHR-based administration task, not a reformulation of a public\nCDI corpus. It is not MIMIC-IV Billing Code, which asks for ICD-10 codes from\ndischarge text, and it is not ClinicReferral (shc_sequoia_med), which is a\nseparate Sequoia clinic referral set. Sibling pages include shc_bmt, shc_conf, shc_ent, shc_gip, shc_privacy,\nshc_proxy, shc_ptbm, shc_sei, and shc_sequoia (ClinicReferral). There is no\nMedHELM family page.\n\n## Saturation and contamination\n\nSaturation is unknown; this page did not obtain a numeric CDI-QA leaderboard\ncell. Contamination risk is low because the item set is private and was built\nfor MedHELM rather than scraped from a public leaderboard. That does not make\nthe labels immune to generic clinical-language overlap, only to wholesale\nmemorization of this CSV.\n\n## How to run it\n\nRun HELM with scenario `shc_cdi_med` and a `data_path` to the private CSV.\nOfficial entries are in `run_entries_medhelm_private_stanford.conf`. Adapter\ninstructions are \"Answer A or B.\" The multiple-choice helper defaults to five\nshots, but the scenario emits only TEST_SPLIT, so HELM's trainer sampler has\nnothing to draw and the protocol is effectively zero-shot. Public reproduction\nof the 1000-item score is not possible without Stanford's file. No matching\ntask was confirmed in lm-evaluation-harness, Inspect Evals, OpenCompass, or\nBIG-bench.\n\n## Reading the numbers\n\nA strong CDI-QA exact-match means the model usually agreed with the gold yes or\nno on these documentation-integrity checks. It does not show that the model can\nwrite a query, code a bill, or catch every clinically unsafe omission. Treat\nthe number as a single-site operations proxy. Compare it with other MedHELM\nadministration tasks (ENT-Referral, HospiceReferral, ClinicReferral) scored the\nsame way, not with ROUGE on note generation.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "shc_cdi",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "shc_cdi_med",
   "CDI-QA"
  ],
  "category": "domain",
  "contamination": {
   "note": "Listed among the 14 private MedHELM datasets that cannot be redistributed because of institutional agreements and patient privacy. Those 14 are not all Stanford Health Care records.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": 1000,
   "size_note": "Nature Medicine Extended Data Table 1 reports 1000 instances evaluated. HELM reads a private filtered CSV (medhelm-CDI-dataset_filtered.csv) with prompt, context, and label columns and tags every row as test.\n",
   "splits": "test only (HELM TEST_SPLIT); no published train or validation split",
   "url": ""
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-001 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-001"
  },
  "harness": {
   "bigbench": "",
   "helm": "shc_cdi_med",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "shc_cdi",
  "last_updated": "2025-08",
  "leaderboard_url": "https://crfm.stanford.edu/helm/medhelm/latest/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "shc_bmt",
    "shc_conf",
    "shc_ent",
    "shc_gip",
    "shc_privacy",
    "shc_proxy",
    "shc_ptbm",
    "shc_sei",
    "shc_sequoia"
   ]
  },
  "measures": "CDI-QA tests whether a model can use an English inpatient note to answer a clinical documentation integrity (CDI) verification question, such as whether a named condition is supported by the chart. HELM places the task under Administration and Workflow, care coordination and planning. The census id is shc_cdi; the runnable HELM scenario is shc_cdi_med.\n",
  "metric": {
   "baseline_note": "Main metric is exact_match on the test split. Class balance and a human CDI specialist baseline were not published for this private set.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1.0,
   "name": "exact_match",
   "random_baseline": null,
   "unit": "0-1 scale"
  },
  "name": "CDI-QA",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2505.23802",
   "title": "Holistic evaluation of large language models for medical tasks with MedHELM",
   "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC13267972/",
   "year": 2026
  },
  "publisher": {
   "authors": [
    "Suhana Bedi",
    "Hejie Cui",
    "Miguel Fuentes",
    "Alyssa Unell",
    "Nigam H. Shah"
   ],
   "org": "Stanford Health Care and Stanford CRFM (MedHELM)",
   "url": "https://crfm.stanford.edu/helm/medhelm/latest/"
  },
  "released": "2025-03",
  "repo_url": "https://github.com/stanford-crfm/helm",
  "saturation": {
   "as_of": "",
   "note": "No dated numeric top exact-match for CDI-QA was recovered from a parsed leaderboard table. The paper heatmap is an image, not a numeric dump.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM shc_cdi_scenario.py (SHCCDIMedScenario, CDI-QA)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/shc_cdi_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM medhelm_run_specs.py (shc_cdi_med run spec)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/run_specs/medhelm_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM run_entries_medhelm_private_stanford.conf",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/presentation/run_entries_medhelm_private_stanford.conf"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM schema_medhelm.yaml (CDI-QA group)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/static/schema_medhelm.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM arXiv:2505.23802",
    "url": "https://arxiv.org/abs/2505.23802"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM Nature Medicine author manuscript (PMC13267972)",
    "url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC13267972/"
   },
   {
    "accessed": "2026-09-08",
    "title": "MedHELM documentation",
    "url": "https://crfm-helm.readthedocs.io/en/latest/medhelm/"
   },
   {
    "accessed": "2026-09-08",
    "title": "stanford-crfm/helm Apache License 2.0",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/LICENSE"
   }
  ],
  "status": "active",
  "subcategory": "clinical documentation integrity verification from notes",
  "summary": "Private Stanford Health Care MedHELM task: verify from a hospital note whether a documented clinical condition is supported, answering A for yes or B for no.\n",
  "tags": [
   "medical",
   "clinical",
   "classification",
   "private-dataset",
   "medhelm",
   "stanford-health-care",
   "documentation"
  ],
  "task_format": "Joint multiple-choice generation. HELM fills a template with the CSV question and note and requires A for yes or B for no, with no extra text.\n"
 }
}