OpenCompass's own cloud-hosted, actively versioned Chinese medical benchmark (v1-v5), covering LLM, multimodal and clinical-agent tracks; documented here as distinct from an unrelated, same-named 2023 benchmark.
unassessed
| Category | domain |
|---|---|
| Subcategory | Chinese medical LLM, multimodal-model and clinical-agent benchmarking platform |
| Page status | active |
| Metric | task-specific: accuracy (exam/multiple-choice subsets), entity- or relation-level F1 (information-extraction subsets), or an LLM-judged 0-100 score (open-ended clinical subsets); no single published formula combines these into one number |
| Direction | higher_is_better |
| Unit | mixed (%, F1, and a 0-100 judged score depending on subset) |
| Dataset size | 700000 |
| Publisher | Shanghai Artificial Intelligence Laboratory (OpenCompass), with co-authors across multiple Chinese hospitals, medical schools and universities |
MedBench is a large, continuously developed Chinese medical evaluation platform, not a single fixed dataset. At its July 2024 launch it assembled what its authors call the largest Chinese medical LLM evaluation set to date (300,901 questions across 43 clinical specialties) and layered on a standardized, cloud-based infrastructure that physically separates questions from ground truth, plus a dynamic evaluation mechanism meant to resist shortcut learning and answer memorization. Later versions expanded scope sharply: MedBench v4 (November 2025) covers over 700,000 expert-curated tasks across 24 primary and 91 secondary specialties with separate tracks for LLMs, multimodal models and clinical agents, reviewed by clinicians from more than 500 institutions; MedBench v5 (June 2026) adds process-oriented auditing and hallucination-propagation tracking for clinical multimodal and agent systems. This page documents the platform actually loaded by OpenCompass's own `MedBench` dataset config -- see Lineage for a naming collision with a different, unrelated Chinese medical benchmark that also calls itself "MedBench."
A multi-track cloud evaluation covering, on the LLM side, five dimensions (medical knowledge QA, medical language understanding, medical language generation, complex medical reasoning, and medical safety/ethics) across dozens of named subsets; separate multimodal and agent-environment tracks were added in v4. Item formats mix multiple-choice exam questions (scored by exact-match accuracy), structured information-extraction tasks such as named-entity and relation extraction (scored by entity- or relation-level F1), and open-ended clinical-dialogue or report-generation tasks (scored by an LLM-as-judge the authors report is calibrated against human ratings).
No model card in ModelSpec reports this benchmark yet.