{
 "body": "\n## What it measures\n\nACI-Bench targets one of the most time-consuming parts of a clinical visit: writing up the\nencounter as a structured note afterward. Given a transcribed doctor-patient conversation, a model\nmust produce a note with the standard sections a clinician expects -- history of present illness,\nphysical exam, results, and assessment and plan -- capturing what was actually said and organising\nit the way a chart note is organised, rather than simply summarising the conversation in prose.\n\nThe dialogues come from three related workflows: plain ambient doctor-patient conversation (the\n`aci` subset), conversation containing spoken triggers meant to invoke a virtual-assistant feature\n(`virtassist`), and conversation that opens with a short doctor dictation before the patient\nexchange begins (`virtscribe`). Because the transcripts come from real or simulated encounters\nprocessed through automatic speech recognition, some retain ASR artefacts -- including, in places,\nswapped `[doctor]`/`[patient]` speaker tags -- which the authors left uncorrected on purpose, to keep\nthe benchmark realistic rather than clean.\n\n## How it is scored\n\nThere is no single official ACI-Bench metric. The dataset originally powered Task B (dialogue-to-\nnote) and Task C (note-to-dialogue and dialogue-to-note, reusing the same pairs) of the MEDIQA-Chat\nand MEDIQA-Sum 2023 shared tasks, which scored submissions with an ensemble of ROUGE, BERTScore and\nBLEURT selected by the organisers to correlate with human judgment of summary quality. HELM's later\nMedHELM implementation takes a different approach entirely: it prompts the model zero-shot to\ngenerate the four-section note (an average response of about 619 tokens, per HELM's own adapter\nconfiguration), then has several annotator language models independently rate the generated note\nagainst the gold note on accuracy, completeness and clarity, each on a 1-5 scale, described as an\n\"LLM jury\" and averaged into HELM's `aci_bench_accuracy` metric -- a HELM naming convention, not a\nliteral exact-match accuracy. These are materially different scoring methods, and a reported\nACI-Bench number should always be checked against which one produced it before being compared to\nanother.\n\n## Dataset and licence\n\nThe dataset totals 207 dialogue-note pairs, confirmed against the reference repository's own stated\nsplits: 67 for training, 20 for validation, and 40 each in three test sets (test1 was MEDIQA-Chat\n2023's Task B test set; test2 and test3 were MEDIQA-Chat and MEDIQA-Sum 2023's respective Task C\ntest sets, reusing dialogue-note pairs from the same underlying collection). The GitHub repository\nstates the data is published under a CC BY 4.0 licence. A third-party Hugging Face mirror\n(`mkieffer/ACI-Bench`) republishes the same splits in parquet form, along with a pointer to\ncorrection scripts for the known ASR speaker-tag-swap issue.\n\n## Who publishes it\n\nACI-Bench was published by Wen-wai Yim, Yujuan Fu, Asma Ben Abacha, Neal Snider, Thomas Lin and\nMeliha Yetisgen in Nature Scientific Data in 2023. Two of the listed authors' contact details in the\nreference repository (Yim and Ben Abacha) are Microsoft email addresses, confirming Microsoft as a\npublishing organisation; this page did not independently confirm other authors' institutional\naffiliations and does not assert them. The reference repository, `wyim/aci-bench`, remains actively\nmaintained (last updated January 2026 at the time of this research) and is the canonical source for\ndata, licence and splits.\n\n## Lineage\n\nACI-Bench has no predecessor benchmark and no successor tracked in this repository. It is one\ndataset shared across two related but distinct shared tasks -- MEDIQA-Chat 2023 (paired with the\nseparate MTS-Dialog dataset for its short-dialogue Task A) and MEDIQA-Sum 2023 -- and is now also\none of several clinical-documentation scenarios inside HELM's broader MedHELM suite, alongside\nMTSamples, MedAlign, DischargeMe and others not covered by this page.\n\n## Saturation and contamination\n\nNo maintained public leaderboard tracking frontier-model ACI-Bench scores over time was found, and\nthe two incompatible scoring methods in use (the original ROUGE/BERTScore/BLEURT ensemble vs.\nHELM's LLM-jury score) make a single saturation read across sources unreliable, so saturation status\nis not established here.\n\nContamination risk is medium: the full dataset, including gold notes, has been publicly available\nvia GitHub and Figshare since the 2023 shared tasks, with no held-out or rotating split maintained\nfor ongoing evaluation, so a model trained on general web or code-hosting data since then has a\nplausible chance of having seen these exact transcript-note pairs.\n\n## How to run it\n\nThe reference data and challenge-format splits live in `wyim/aci-bench` on GitHub. HELM implements\nit as the `aci_bench` scenario inside its MedHELM suite: a zero-shot generation task instructing the\nmodel to produce a four-section note, scored by an LLM jury defined in `ACIBenchAnnotator`\n(accuracy, completeness and clarity, each 1-5, averaged across annotator models). No\nlm-evaluation-harness, inspect_evals, OpenCompass or BIG-bench implementation was confirmed. Because\nthe original MEDIQA shared-task scoring (ROUGE/BERTScore/BLEURT) and HELM's LLM-jury scoring measure\ndifferent things on different scales, results from the two should never be compared directly.\n\n## Reading the numbers\n\nA strong ACI-Bench score means a model reliably pulls the clinically relevant content out of a\nmessy, real-world-style conversation transcript and organises it into the section structure\nclinicians expect -- a genuinely useful proxy for ambient clinical documentation tools. It does not\nmean the note is safe to use unreviewed: neither scoring method directly penalises a clinically\ndangerous omission or fabrication the way a domain expert reviewer would, and the dataset's own\nretained ASR errors mean part of what is being tested is robustness to imperfect input, not just\nsummarisation quality. Because ROUGE/BERTScore/BLEURT and LLM-jury scores are not on the same scale\nor measuring the same thing, always check which scoring method underlies a reported number, and\ntreat any single score as a proxy to be checked against qualitative note review rather than a\nstand-alone clinical-safety signal.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "aci_bench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "ACI-BENCH"
  ],
  "category": "domain",
  "contamination": {
   "note": "The full dataset, including gold notes, has been publicly available on GitHub and via Figshare since the 2023 MEDIQA-Chat and MEDIQA-Sum shared tasks, with no held-out or rotating split described for ongoing evaluation, so exact-pair memorisation is plausible for any model trained on general web or GitHub-derived data since then.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC BY 4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 207,
   "size_note": "207 total dialogue-note pairs across five splits, per the reference repository's own published counts: train 67, valid 20, test1 40 (the MEDIQA-Chat 2023 Task B test set), test2 40 (the MEDIQA-Chat 2023 Task C test set), test3 40 (the MEDIQA-Sum 2023 Task C test set). The collection is further divided into three sub-workflows: aci (ambient doctor-patient dialogue, the largest subset), virtassist (dialogue with spoken triggers for a virtual-assistant feature), and virtscribe (dialogue with a short doctor dictation at the start).\n",
   "splits": "train (67), valid (20), test1 (40), test2 (40), test3 (40)",
   "url": "https://github.com/wyim/aci-bench"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 2, slice D",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "aci_bench (part of the MedHELM suite; zero-shot generation, LLM-jury scoring)",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "aci_bench",
  "last_updated": "2026-01",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "ACI-Bench (Ambient Clinical Intelligence Benchmark) tests a model's ability to convert a transcribed doctor-patient conversation into a structured clinical visit note -- the documentation task physicians otherwise do by hand after (or during) every visit. Given a dialogue transcript, the model must produce a note covering sections such as history of present illness, physical exam findings, results, and assessment and plan. It is single-turn, text-to-text, English-language summarisation grounded in real (de-identified or simulated) clinical encounters rather than synthetic dialogue.\n",
  "metric": {
   "baseline_note": "No single official metric exists. The MEDIQA-Chat/MEDIQA-Sum shared tasks that used this dataset scored submissions with an ensemble of ROUGE, BERTScore and BLEURT chosen to correlate with human judgment of note quality. HELM's MedHELM implementation instead uses a documented \"LLM jury\": several annotator models each rate a generated note against the reference on accuracy, completeness and clarity, each on a 1-5 scale, averaged into a single \"Jury Score\" (HELM calls this metric aci_bench_accuracy, a naming choice from HELM's own schema rather than a correctness/accuracy metric in the usual sense). These two scoring approaches are not comparable, so a reported ACI-Bench number should be checked for which one produced it.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "varies by reporter: ROUGE / BERTScore / BLEURT ensemble (original MEDIQA shared tasks) or LLM-jury score (HELM's MedHELM implementation)",
   "random_baseline": null,
   "unit": "points (scale depends on method)"
  },
  "name": "ACI-Bench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "Aci-bench: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation",
   "url": "https://www.nature.com/articles/s41597-023-02487-3",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Wen-wai Yim",
    "Yujuan Fu",
    "Asma Ben Abacha",
    "Neal Snider",
    "Thomas Lin",
    "Meliha Yetisgen"
   ],
   "org": "Microsoft",
   "url": "https://github.com/wyim/aci-bench"
  },
  "released": "2023",
  "repo_url": "https://github.com/wyim/aci-bench",
  "saturation": {
   "as_of": "",
   "note": "No maintained public leaderboard tracking frontier-model scores over time was found. HELM's MedHELM suite reports current model runs on its own site using the LLM-jury metric, but this page did not independently confirm a current top score there, and the original shared-task leaderboards (MEDIQA-Chat / MEDIQA-Sum 2023) reflect submissions from that specific 2023 competition rather than an ongoing tracker.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Aci-bench: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation (Yim et al., Nature Scientific Data, 2023)",
    "url": "https://www.nature.com/articles/s41597-023-02487-3"
   },
   {
    "accessed": "2026-09-08",
    "title": "wyim/aci-bench GitHub repository (README, data splits, licence)",
    "url": "https://github.com/wyim/aci-bench"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM aci_bench_scenario.py (ACIBenchScenario, MedHELM suite)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/aci_bench_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM ACIBenchAnnotator (LLM-jury scoring prompt and rubric)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/annotation/aci_bench_annotator.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "mkieffer/ACI-Bench dataset card, Hugging Face (third-party mirror with split counts)",
    "url": "https://huggingface.co/datasets/mkieffer/ACI-Bench"
   }
  ],
  "status": "active",
  "subcategory": "clinical note generation from doctor-patient dialogue",
  "summary": "Tests whether a model can turn a doctor-patient conversation transcript into a structured clinical note; 207 real dialogue-note pairs, the largest public dataset of its kind at publication.",
  "tags": [
   "medical",
   "clinical",
   "summarization",
   "dialogue",
   "domain"
  ],
  "task_format": "Given a full doctor-patient dialogue transcript as input, the model generates free text structured as a clinical note with the required section headers. Some transcripts include automatic-speech-recognition artefacts, including occasional swapped speaker tags ([doctor]/[patient]), which the dataset authors left uncorrected deliberately to reflect a realistic, imperfect transcription pipeline rather than clean input.\n"
 }
}