{
 "body": "\n## What it measures\n\nThe name \"ClinicBench\" names two different things, and the OpenCompass task this page documents runs only the smaller of them. OpenCompass's `ClinicBench` dataset config loads a Hugging Face dataset called `xuxuxuxuxu/Pharmacology-QA`: 213 four-option multiple-choice pharmacology questions, contributed to OpenCompass in May 2025 alongside PubMedQA and ScienceQA, with no citation, licence or description attached to the mirror itself beyond its raw column schema. Matching its exact row count (213) and subject against the wider literature traces this data to one specific task, \"Pharmacology QA for Emerging Drugs,\" inside a much larger benchmark suite also called ClinicBench, published by Fenglin Liu and co-authors (University of Oxford and Amazon) at EMNLP 2024. That original ClinicBench spans three scenarios, eleven tasks and seventeen datasets -- over 20,000 test samples in total -- covering clinical language generation, understanding and reasoning, evaluated across twenty-two LLMs, plus a human expert evaluation of clinical usefulness. OpenCompass implements only this one 213-item pharmacology task from that suite, without stating the connection anywhere in its own code.\n\nThe task asks a model to answer a multiple-choice pharmacology question about drugs released between October 2023 and April 2024, derived from DrugBank -- material the original authors chose specifically because it postdated the training cutoffs of the models they were evaluating.\n\n## How it is scored\n\nThe model is prompted to think step by step and end its response with a line of the form \"ANSWER: $LETTER\". An LLM judge compares the extracted letter against the gold label under OpenCompass's `GenericLLMEvaluator`, following grading instructions general enough to also handle multi-select and fill-in-the-blank answers, though this task is a single-answer, four-option question. With four options, a naive random baseline is 25%. The prompt template's own instruction text describes options as \"one of ABCDEFGHIJKLMNOP,\" but that phrasing is shared boilerplate reused across several OpenCompass multiple-choice tasks, not evidence this dataset offers more than the four options (A-D) actually present in its columns.\n\n## Dataset and licence\n\nThe 213-question count is confirmed directly from the Hugging Face dataset OpenCompass loads. That dataset carries no licence field, no description, and no citation back to any paper; it was uploaded by a Hugging Face account matching the name pattern of the OpenCompass contributor who added the `ClinicBench` config in the same pull request as PubMedQA and ScienceQA. The strongest evidence connecting it to Liu et al.'s suite is that Table 1 of that paper lists a task named \"Pharmacology QA for Emerging Drugs,\" \"derived from DrugBank,\" with exactly 213 items scored by accuracy -- an exact match on both count and description. The original GitHub repository (`AI-in-Health/ClinicBench`) is Apache-2.0 licensed and describes the paper's EMNLP 2024 retitling as \"Large Language Models Are Poor Clinical Decision-Makers: A Comprehensive Benchmark,\" different from the arXiv preprint's title. Whether that licence extends to this derived 213-row mirror, and what DrugBank's own terms permit, are not established here.\n\n## Who publishes it\n\nThe wider ClinicBench suite this task is drawn from was built by Fenglin Liu, Zheng Li, Hongjian Zhou and co-authors at the University of Oxford's Institute of Biomedical Engineering and Amazon, with further co-authors from Harvard T.H. Chan School of Public Health and Institut Polytechnique de Paris, posted to arXiv in April 2024 and published at EMNLP 2024. The 213-item mirror OpenCompass actually runs was instead re-packaged and uploaded independently to Hugging Face and OpenCompass by a different contributor, with no visible connection back to Liu et al.'s team; whether that re-packaging was authorised is not established.\n\n## Lineage\n\nThis is a case where a harness id points to a narrow fragment of a much better-known, much larger project sharing its exact name. Liu et al.'s full ClinicBench aggregates eleven existing clinical datasets (including MedQA and MedMCQA, both separately documented elsewhere in this repository, plus MMLU-Med, which is not) with six the authors built themselves, of which \"Pharmacology QA for Emerging Drugs\" is one; the other five novel datasets (open-ended clinical decision-making, long document processing, drug-interaction analysis) are not implemented here and have no page in this repository. Anyone citing a \"ClinicBench\" score should specify whether it comes from this narrow task or from Liu et al.'s full suite run through their own code, since the two are not comparable.\n\n## Saturation and contamination\n\nNo score for this 213-item task, under either name, was found published in any leaderboard, paper or model card, so saturation is unknown. Contamination risk sits at medium: the source questions were deliberately drawn from drugs released between October 2023 and April 2024, so their content would postdate the evaluated models' training cutoffs at the time -- a design meant to keep contamination low at release. That protection erodes for any model trained later: by this research date those \"emerging\" drugs are over two years old and plausibly present in newer training data, and the mirror carries the full question set and gold labels with no gating.\n\n## How to run it\n\nOpenCompass implements this as the `ClinicBench` dataset, loading `xuxuxuxuxu/Pharmacology-QA` and grading with `GenericLLMEvaluator`. No lm-evaluation-harness, inspect_evals, HELM or BIG-bench implementation of either the narrow OpenCompass task or the full Liu et al. suite was found. Reproducing Liu et al.'s original evaluation (three scenarios, eleven tasks, seventeen datasets, twenty-two models) requires the authors' own code at `github.com/AI-in-Health/ClinicBench`, which OpenCompass's task does not draw on.\n\n## Reading the numbers\n\nBefore treating any \"ClinicBench\" figure as meaningful, establish which thing it measures: this page's narrow, 213-item pharmacology slice, or Liu et al.'s much broader seventeen-dataset suite run through the original authors' own code. A score from the former says something specific and narrow -- whether a model can answer multiple-choice questions about a fixed set of drugs from late 2023/early 2024 -- and, given the task's design, an especially high score from a model with a later training cutoff may reflect memorised drug facts rather than clinical reasoning. It says nothing about the generation, understanding or open-ended decision-making tasks that make up most of the original suite, and neither this page nor OpenCompass's implementation currently covers those.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "clinicbench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Pharmacology QA for Emerging Drugs"
  ],
  "category": "domain",
  "contamination": {
   "note": "The source task was deliberately built from drugs released between October 2023 and April 2024, specifically so its content would postdate the training cutoffs of the models Liu et al. evaluated at the time -- a design meant to keep contamination low at release. That protection erodes for any model with a later training cutoff, and by this research date (September 2026) those \"emerging\" drugs are over two years old and plausibly present in newer models' pretraining data. The OpenCompass-hosted mirror also carries the full question set and gold labels with no gating or canary string.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 213,
   "size_note": "213 questions, confirmed via the Hugging Face datasets-server against the mirror OpenCompass actually loads (xuxuxuxuxu/Pharmacology-QA). This exact count and a \"Predict the correct answer to the given pharmacology question for the new drugs released between October 2023 and April 2024... Derived from DrugBank\" description both match one row of Table 1 in Liu et al.'s ClinicBench paper, which is the strongest evidence tying this mirror back to that paper's \"Pharmacology QA for Emerging Drugs\" task; the mirror itself carries no citation confirming this directly. The wider ClinicBench suite that task comes from totals seventeen datasets and over 20,000 test samples, none of the rest of which are wired into OpenCompass's ClinicBench config.\n",
   "splits": "single unsplit list of 213 questions (Hugging Face 'train'); no held-out test partition",
   "url": "https://huggingface.co/datasets/xuxuxuxuxu/Pharmacology-QA"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 5, slice D",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "ClinicBench",
   "other": ""
  },
  "id": "clinicbench",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "The name \"ClinicBench\" names two different things, and the OpenCompass task this page documents runs only the smaller of them. OpenCompass's `ClinicBench` dataset config loads a Hugging Face dataset, xuxuxuxuxu/Pharmacology-QA: 213 four-option multiple-choice pharmacology questions, with no citation, licence or description attached to the mirror itself. Matching its exact row count and task description against the wider literature traces this data to one specific task, \"Pharmacology QA for Emerging Drugs,\" inside a much larger benchmark suite also called ClinicBench, published by Fenglin Liu and co-authors (University of Oxford and Amazon) at EMNLP 2024. That original ClinicBench spans three scenarios, eleven tasks and seventeen datasets (over 20,000 test samples in total) covering clinical language generation, understanding and reasoning, evaluated across twenty-two LLMs -- of which OpenCompass implements only this one 213-item pharmacology task, without stating that connection anywhere in its own code.\n",
  "metric": {
   "baseline_note": "Four-option multiple choice gives a naive random baseline of 25%. No score for this specific OpenCompass task, under either name, was found published in any leaderboard or paper reviewed for this page, so no top or human baseline is recorded.\n",
   "direction": "higher_is_better",
   "max_score": 100,
   "name": "accuracy (LLM-judged consistency with the gold letter)",
   "random_baseline": 25,
   "unit": "%"
  },
  "name": "ClinicBench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2405.00716",
   "title": "Large Language Models in the Clinic: A Comprehensive Benchmark",
   "url": "https://arxiv.org/abs/2405.00716",
   "year": 2024
  },
  "publisher": {
   "authors": [],
   "org": "",
   "url": ""
  },
  "released": "2024-04",
  "repo_url": "https://github.com/AI-in-Health/ClinicBench",
  "saturation": {
   "as_of": "",
   "note": "No score for OpenCompass's 213-item ClinicBench task, under either name, was found published in any leaderboard, paper or model card reviewed for this page.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass ClinicBench dataset config (loads xuxuxuxuxu/Pharmacology-QA)",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/ClinicBench/ClinicBench_llmjudge_gen_d09668.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass ClinicBenchDataset loader",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/datasets/ClinicBench.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "xuxuxuxuxu/Pharmacology-QA dataset metadata, Hugging Face (213 rows, A/B/C/D/question/choices/label columns, no description)",
    "url": "https://huggingface.co/api/datasets/xuxuxuxuxu/Pharmacology-QA"
   },
   {
    "accessed": "2026-09-08",
    "title": "Add ClinicBench, PubMedQA and ScienceQA (#2061), stanford-crfm/helm -- opencompass PR by xuxuxuxuxuxjh",
    "url": "https://github.com/open-compass/opencompass/pull/2061"
   },
   {
    "accessed": "2026-09-08",
    "title": "Large Language Models in the Clinic: A Comprehensive Benchmark (Liu et al.)",
    "url": "https://arxiv.org/abs/2405.00716"
   },
   {
    "accessed": "2026-09-08",
    "title": "ClinicBench paper, full text incl. Table 1 (ar5iv)",
    "url": "https://ar5iv.labs.arxiv.org/html/2405.00716"
   },
   {
    "accessed": "2026-09-08",
    "title": "AI-in-Health/ClinicBench repository metadata (Apache-2.0; EMNLP 2024 retitled description)",
    "url": "https://api.github.com/repos/AI-in-Health/ClinicBench"
   }
  ],
  "status": "unknown",
  "subcategory": "pharmacology multiple-choice QA for emerging drugs (a single task from a larger 17-dataset clinical suite)",
  "summary": "OpenCompass's ClinicBench task runs only a 213-item pharmacology-QA slice of a much larger, unrelated-looking 17-dataset clinical benchmark suite published under the same name.",
  "tags": [
   "domain",
   "clinical",
   "pharmacology",
   "drugbank",
   "name-collision"
  ],
  "task_format": "Four-option multiple-choice pharmacology question (options labelled A-D in the source data); the model is asked to think step by step and give a final \"ANSWER: $LETTER\" line. The prompt template's own instruction text mentions options \"one of ABCDEFGHIJKLMNOP\", but that is boilerplate shared across several OpenCompass multiple-choice tasks, not evidence this dataset has more than four options.\n"
 }
}