HealthBench grades a model's responses in realistic, multi-turn health conversations against physician-written rubrics, using another model as the judge.
unassessed
| Category | domain |
|---|---|
| Subcategory | multi-turn health conversations, graded by a model against physician-written rubrics |
| Page status | active |
| Metric | rubric criteria met, normalized per conversation (mean score, clipped to [0,1]) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 5000 |
| Dataset licence | MIT for the evaluation code and released conversation/rubric data, per the openai/simple-evals GitHub repository. The paper separately asks that examples not be reposted online in plain text or images, to reduce the chance of leakage into future training corpora, and ships a canary string for filtering -- a usage request layered on top of, not a replacement for, the MIT code licence. |
| Publisher | OpenAI |
HealthBench tests how a model responds across 5,000 realistic health conversations that stand in for a layperson, caregiver or clinician, produced through both synthetic generation and human adversarial testing so they are multi-turn (1 to 19 turns, averaging 2.6) and multilingual rather than single-shot trivia questions. Conversations are organised into seven themes -- emergency referrals, context seeking, global health, health data tasks, expertise-tailored communication, responding under uncertainty, and response depth -- each probing a different way a health-related answer can go right or wrong, from missing an emergency to using jargon with a layperson.
The model reads a multi-turn conversation and must produce the best possible final response to the user's last message. There is no fixed answer to match; every conversation instead carries its own physician-written rubric of scoring criteria that a grading model checks the response against.
No model card in ModelSpec reports this benchmark yet.