Tests whether a model can turn a doctor-patient conversation transcript into a structured clinical note; 207 real dialogue-note pairs, the largest public dataset of its kind at publication.
unassessed
| Category | domain |
|---|---|
| Subcategory | clinical note generation from doctor-patient dialogue |
| Page status | active |
| Metric | varies by reporter: ROUGE / BERTScore / BLEURT ensemble (original MEDIQA shared tasks) or LLM-jury score (HELM's MedHELM implementation) |
| Direction | higher_is_better |
| Unit | points (scale depends on method) |
| Dataset size | 207 |
| Dataset licence | CC BY 4.0 |
| Publisher | Microsoft |
ACI-Bench (Ambient Clinical Intelligence Benchmark) tests a model's ability to convert a transcribed doctor-patient conversation into a structured clinical visit note -- the documentation task physicians otherwise do by hand after (or during) every visit. Given a dialogue transcript, the model must produce a note covering sections such as history of present illness, physical exam findings, results, and assessment and plan. It is single-turn, text-to-text, English-language summarisation grounded in real (de-identified or simulated) clinical encounters rather than synthetic dialogue.
Given a full doctor-patient dialogue transcript as input, the model generates free text structured as a clinical note with the required section headers. Some transcripts include automatic-speech-recognition artefacts, including occasional swapped speaker tags ([doctor]/[patient]), which the dataset authors left uncorrected deliberately to reflect a realistic, imperfect transcription pipeline rather than clean input.
No model card in ModelSpec reports this benchmark yet.