{ "body": "\n## What it measures\n\nMedDialog evaluates medical dialogue summarization. A model reads a conversation between a patient and doctor and generates a concise summary of the central question or exchange. HELM describes English data from HealthCareMagic and iCliniq.\n\nThe two sources have different dialogue styles. The scenario notes that HealthCareMagic summaries are more abstractive and formal, while iCliniq summaries are more patient-written.\n\n## How it is scored\n\nHELM names `med_dialog_accuracy` as its main metric and uses the generated test summary as the correct reference. The scenario does not define the metric formula or normalization, so scores should not be interpreted as a particular ROUGE, BLEU, or semantic metric without the active HELM evaluator configuration.\n\n## Dataset and licence\n\nHELM describes about 0.26 million English dialogues. Its cited preprocessing table gives HealthCareMagic 181,122 train, 22,641 validation, and 22,642 test examples; iCliniq has 24,851 train, 3,105 validation, and 3,108 test examples. The source says the raw dialogues come from healthcaremagic.com and icliniq.com and that their copyrights belong to those sites. A redistribution licence was not established.\n\n## Who publishes it\n\nThe benchmark paper is by Chen and colleagues, published as arXiv:2004.03329. UCSD AI4H previously maintained the linked code repository, but as of 2026-09-08 `github.com/UCSD-AI4H/Medical-Dialogue-System` returns a 404 rather than redirecting; the account no longer resolves. HELM provides a scenario that downloads the test data directly from a CodaLab bundle, independent of that repository. No current standalone leaderboard was established.\n\n## Lineage\n\nMedDialog is a standalone dataset with HealthCareMagic and iCliniq subsets. The sources do not establish a successor benchmark.\n\n## Saturation and contamination\n\nSaturation is unknown. The dialogues originate from public healthcare websites, so exposure is plausible, but no model-training analysis was found. Copyright and privacy considerations also make dataset handling material to reproducibility.\n\n## How to run it\n\nHELM\u2019s `med_dialog` scenario takes a `healthcaremagic` or `icliniq` subset, downloads its test JSON, and creates dialogue-to-summary instances. It uses the test split only for the zero-shot scenario. Record the subset and active HELM metric implementation.\n\n## Reading the numbers\n\nA strong score indicates concise content preservation on these medical conversations. It does not establish clinical safety, factual correctness, or suitability for patient communication. Compare subsets separately because their writing styles and summary properties differ.\n", "build": { "built_at": "2026-09-09T16:56:50+00:00", "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11", "eligibility_as_of": "2026-09-09" }, "disposition": { "canonical_id": "med_dialog", "reasons": [], "status": "unassessed", "verified_results": [] }, "models_covered": [], "page": { "aliases": [], "category": "domain", "contamination": { "but current model exposure is not established.": null, "note": "The source describes public web-origin dialogues", "risk": "medium" }, "dataset": { "languages": [ "English" ], "license": "", "modalities": [ "text" ], "public_test_set": true, "size": 260000, "size_note": "HELM describes about 0.26 million English dialogues; its cited preprocessing gives HealthCareMagic 226,405 and iCliniq 31,064 examples across train, validation, and test.", "splits": "train", "test": null, "url": "https://github.com/UCSD-AI4H/Medical-Dialogue-System", "validation": null }, "freshness": { "luna-batch-057 (Codex coordinated)": null, "researched": "2026-09-08", "researched_by": "GPT-5.6 Luna", "reviewed": "2026-09-08", "reviewed_by": "Claude Sonnet 5 independent review, luna-batch-057" }, "harness": { "bigbench": "", "helm": "med_dialog", "inspect_evals": "", "lm_eval": "", "opencompass": "", "other": "" }, "id": "med_dialog", "last_updated": "", "leaderboard_url": "", "lineage": { "family": "", "predecessor": "", "successors": [], "variants": [ "med_dialog_healthcaremagic", "med_dialog_icliniq" ] }, "measures": "MedDialog presents a medical dialogue and asks a model to produce a one-sentence summary of the patient question or exchange. It covers English conversations between patients and doctors.", "metric": { "baseline_note": "HELM names the metric but does not define its formula in the scenario.", "direction": "higher_is_better", "human_baseline": null, "max_score": null, "name": "med_dialog_accuracy", "random_baseline": null, "unit": "" }, "name": "MedDialog", "page_kind": "benchmark", "paper": { "arxiv": "2004.03329", "title": "MedDialog: a large-scale medical dialogue dataset", "url": "https://arxiv.org/abs/2004.03329", "year": 2020 }, "publisher": { "authors": [ "Shu Chen", "Zeqian Ju", "Xiangyu Dong", "Hongchao Fang", "Sicheng Wang", "Yue Yang", "Jiaqi Zeng", "Ruisi Zhang", "Ruoyu Zhang", "Meng Zhou", "Penghui Zhu", "Pengtao Xie" ], "org": "UCSD AI4H / MedDialog authors", "url": "https://github.com/UCSD-AI4H/Medical-Dialogue-System" }, "released": "2020", "repo_url": "https://github.com/UCSD-AI4H/Medical-Dialogue-System", "saturation": { "as_of": "", "note": "No current authoritative leaderboard was established.", "status": "unknown", "top_score": null }, "sources": [ { "accessed": "2026-09-08", "title": "HELM MedDialog scenario", "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/med_dialog_scenario.py" }, { "accessed": "2026-09-08", "title": "MedDialog paper", "url": "https://arxiv.org/abs/2004.03329" }, { "accessed": "2026-09-08", "title": "MedDialog repository", "url": "https://github.com/UCSD-AI4H/Medical-Dialogue-System" } ], "status": "active", "subcategory": "medical dialogue summarization", "summary": "MedDialog evaluates concise summaries of English doctor-patient conversations from HealthCareMagic and iCliniq.", "tags": [ "medical", "dialogue", "summarization" ], "task_format": "Dialogue input followed by a reference summary; generate a concise medical summary." } }