{
 "body": "\n## What it measures\n\nMedHELM Configurable is a scenario wrapper for biomedical evaluations. It does not identify one fixed question set. A configuration supplies a description, CSV dataset, prompt file, and metric definitions, allowing different medical tasks to run through a common HELM path.\n\nThe scenario supports templates populated from dataset columns and can include correct and incorrect answer references. Its language and clinical domain therefore depend on the selected configuration.\n\n## How it is scored\n\nThe configuration names the main metric and can list additional metrics. The wrapper supports ordinary reference-based grading and a `jury_score` path for LLM-as-judge evaluations. There is no single metric, maximum, or baseline for the identifier as a whole.\n\n## Dataset and licence\n\nEach configuration points to its own CSV dataset and prompt file. The wrapper requires a `correct_answer` column unless `jury_score` is the sole metric, and parses `incorrect_answers` as JSON when present. Aggregate size and licence are therefore configuration-specific and not established here.\n\n## Who publishes it\n\nThe wrapper is maintained in Stanford CRFM\u2019s HELM repository. The inspected source does not establish a separate MedHELM paper, author list, or leaderboard for this configurable identifier.\n\n## Lineage\n\nThis is a family or framework page. Individual MedHELM configurations should be represented separately when their dataset and metric definitions are known. No fixed predecessor or successor was established.\n\n## Saturation and contamination\n\nSaturation and contamination cannot be assessed without selecting a configuration. A public prompt or dataset may have a different exposure profile from a restricted clinical set.\n\n## How to run it\n\nUse HELM\u2019s `medhelm_configurable` scenario with a scenario name and configuration path. It verifies the prompt and dataset files, reads the CSV, fills template fields, and emits test instances. Record the exact configuration, prompt, dataset revision, and metric set.\n\n## Reading the numbers\n\nA score belongs to a particular MedHELM configuration, not to this wrapper in isolation. It may measure answer matching, judging, or another configured objective. Always report the configuration identity and metric definition before comparing results.\n\nThe wrapper\u2019s flexibility makes bare benchmark names ambiguous. A complete citation should include the configuration file, data revision, prompt template, and metric list.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "medhelm_configurable",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "MedHELM"
  ],
  "category": "domain",
  "contamination": {
   "note": "Each configuration has its own data and exposure profile.",
   "risk": "unknown"
  },
  "dataset": {
   "languages": [],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": null,
   "size": null,
   "size_note": "Dataset size is configuration-dependent.",
   "splits": "test",
   "url": "https://github.com/stanford-crfm/helm"
  },
  "freshness": {
   "luna-batch-057 (Codex coordinated)": null,
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, luna-batch-057"
  },
  "harness": {
   "bigbench": "",
   "helm": "medhelm_configurable",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "medhelm_configurable",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "This identifier names a configurable evaluation scenario rather than one fixed dataset. A benchmark configuration supplies a CSV dataset, prompt template, description, and one or more metrics.",
  "metric": {
   "baseline_note": "The metric is selected by each configuration.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "",
   "random_baseline": null,
   "unit": ""
  },
  "name": "MedHELM Configurable",
  "page_kind": "family",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [],
   "org": "Stanford CRFM HELM",
   "url": "https://github.com/stanford-crfm/helm"
  },
  "released": "",
  "repo_url": "https://github.com/stanford-crfm/helm",
  "saturation": {
   "as_of": "",
   "note": "No aggregate leaderboard is meaningful without a selected configuration.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM MedHELM configurable scenario",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/medhelm_configurable_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM MedHELM run specifications",
    "url": "https://github.com/stanford-crfm/helm/tree/main/src/helm/benchmark/run_specs/medhelm"
   }
  ],
  "status": "active",
  "subcategory": "biomedical evaluation framework",
  "summary": "MedHELM Configurable is a HELM scenario wrapper for configurable biomedical datasets, prompts, references, and metrics.",
  "tags": [
   "medical",
   "configurable",
   "evaluation-framework"
  ],
  "task_format": "Configuration-defined biomedical prompt with optional correct and incorrect answers."
 }
}