{
 "body": "\n## What it measures\n\nMEDEC measures medical error detection and correction in clinical notes. Each narrative is either correct or contains an error. Error types in the scenario include diagnosis, management, treatment, pharmacotherapy, and causal organism.\n\nFor an erroneous note, the intended output identifies the sentence and supplies corrected wording. For a correct note, the expected output is `CORRECT`.\n\n## How it is scored\n\nHELM\u2019s main metric is `medec_error_flag_accuracy`, focused on whether the error flag is correct. The scenario encodes a corrected reference as the sentence identifier followed by corrected text. Detection, localization, and correction should be reported separately when possible.\n\n## Dataset and licence\n\nThe scenario describes 3,848 texts: 2,189 MS training, 574 MS plus 160 UW validation, and 597 MS plus 328 UW test. It downloads the MS test CSV at a pinned repository commit for HELM\u2019s test run. A dataset licence was not established from the inspected sources.\n\n## Who publishes it\n\nMEDEC was introduced by Ben Abacha and colleagues and used in the MEDIQA-CORR shared task. The paper is arXiv:2412.19260. The authors maintain the GitHub repository; no current standalone leaderboard was established.\n\n## Lineage\n\nMEDEC is a standalone medical error benchmark associated with MEDIQA-CORR. The inspected sources do not establish a predecessor or successor.\n\n## Saturation and contamination\n\nThe scenario describes recent systems as still trailing medical doctors on error tasks, supporting an open status. Public repository files create medium exposure risk, but no quantitative contamination study was found.\n\n## How to run it\n\nUse HELM\u2019s `medec` scenario. It downloads the pinned MS test CSV, ignores the training file, and emits zero-shot test instances. Record the pinned commit, output format, and whether scores cover only detection or all correction stages.\n\n## Reading the numbers\n\nA high detection score means the model can classify the presence of errors in these narratives. It does not prove that corrections are clinically valid or safe for deployment. Inspect localization, correction fidelity, error type, and MS-versus-UW composition alongside the headline score.\n\nThe expected output couples a sentence identifier with corrected text. Exact matching can penalize clinically equivalent rewrites, so component-level evaluation is needed to separate medical correctness from formatting.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "medec",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "domain",
  "contamination": {
   "note": "The repository exposes benchmark files; model exposure was not quantitatively established.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 3848,
   "size_note": "The scenario describes 2,189 MS training, 734 validation (574 MS and 160 UW), and 925 test (597 MS and 328 UW) texts.",
   "splits": "train",
   "test": null,
   "url": "https://github.com/abachaa/MEDEC",
   "validation": null
  },
  "freshness": {
   "luna-batch-057 (Codex coordinated)": null,
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, luna-batch-057"
  },
  "harness": {
   "bigbench": "",
   "helm": "medec",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "MEDIQA-CORR"
  },
  "id": "medec",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "MEDEC presents a clinical narrative that is either correct or contains an error. A model must identify the error status, locate the erroneous sentence, and provide a correction when applicable.",
  "metric": {
   "baseline_note": "The scenario mentions doctors outperforming recent systems but gives no numeric human baseline.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "medec_error_flag_accuracy",
   "random_baseline": null,
   "unit": "percent"
  },
  "name": "MEDEC",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2412.19260",
   "title": "MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes",
   "url": "https://arxiv.org/abs/2412.19260",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Asma Ben Abacha",
    "et al."
   ],
   "org": "MEDIQA-CORR / MEDEC authors",
   "url": "https://github.com/abachaa/MEDEC"
  },
  "released": "2024",
  "repo_url": "https://github.com/abachaa/MEDEC",
  "saturation": {
   "as_of": "",
   "note": "The scenario reports remaining gaps versus doctors but no current top score.",
   "status": "open",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM MEDEC scenario",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/medec_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "MEDEC paper",
    "url": "https://arxiv.org/abs/2412.19260"
   },
   {
    "accessed": "2026-09-08",
    "title": "MEDEC repository",
    "url": "https://github.com/abachaa/MEDEC"
   }
  ],
  "status": "active",
  "subcategory": "medical error detection and correction",
  "summary": "MEDEC evaluates detection and correction of medical errors in clinical narratives.",
  "tags": [
   "medical",
   "error-detection",
   "correction"
  ],
  "task_format": "Clinical text; output an error sentence identifier and correction, or CORRECT."
 }
}