{
 "body": "\n## What it measures\n\nHealthBench Hard tests how a language model performs in realistic, multi-turn healthcare conversations, standing in for a patient, caregiver or clinician. Each conversation was written or adapted with physician input to probe a specific behaviour: giving correct information, communicating clearly, staying appropriately cautious, or handling an emergency or a piece of context correctly. It is not a separate dataset but a fixed slice of the full HealthBench set.\n\nHealthBench Hard consists of the 1,000 conversations, out of HealthBench's 5,000, that scored lowest on average across a panel of model providers when OpenAI built the benchmark -- the conversations that current frontier models at the time handled worst, rather than a random or topic-based sample.\n\n## How it is scored\n\nEach conversation carries its own physician-written rubric of scoring criteria: some award points for good behaviour, others subtract points for specific failures such as unsafe advice or a missed red flag. A grader model checks a response against every criterion in its conversation's rubric, the matched points are summed and normalized against the criteria's total possible points, and per-conversation scores are clipped to a 0-1 range before being averaged across the sample, with bootstrap resampling used to produce confidence intervals. Because grading depends on an LLM judge applying the rubric, reported scores can shift slightly with the grader model used.\n\n## Dataset and licence\n\nFull HealthBench totals 5,000 conversations built with input from 262 physicians across 60 countries, carrying more than 48,000 individual rubric criteria across the set. HealthBench Hard is the 1,000 of those conversations with the lowest average score across model providers at construction time, selected so difficulty would not be concentrated on one provider's particular weaknesses and so it would not over-represent conversations that were simply hard to grade rather than hard to answer. OpenAI publishes the evaluation code and conversation data through the openai/simple-evals repository on GitHub under the MIT licence; conversations and rubrics are public.\n\n## Who publishes it\n\nHealthBench comes from Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quinonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke and Karan Singhal at OpenAI, published as \"HealthBench: Evaluating Large Language Models Towards Improved Human Health\" on arXiv in May 2025. OpenAI maintains the reference implementation and continues to report HealthBench and HealthBench Hard scores for its own model releases. The UK AI Security Institute's Inspect Evals project also maintains an independent port of the task.\n\n## Lineage\n\nHealthBench Hard is a fixed-difficulty slice of the base HealthBench benchmark rather than a separate dataset, so its natural predecessor is HealthBench itself. It sits alongside HealthBench Consensus, a smaller, physician-agreement-weighted subset defined in the same paper and repository; neither HealthBench nor HealthBench Consensus has a page in this repository within this batch. A mental-health-focused extension, HealthBench-Psych, has since been published as a separate follow-on paper outside OpenAI's original release.\n\n## Saturation and contamination\n\nAt the benchmark's release in May 2025, OpenAI reported that the best-performing model reached only about 32% on HealthBench Hard, far from the ceiling: the subset was deliberately built from the lowest-scoring conversations in a multi-provider baseline so that it would resist quick saturation. No source consulted here gives a more recent top score, so the current standing for 2026-era frontier models is not established. Because the conversations and rubrics are fully public, and OpenAI itself re-reports this benchmark across successive model releases, there is a plausible route for rubric criteria to reach future training data over time, but no source consulted quantifies that risk, so contamination risk is not established either.\n\n## How to run it\n\nThe UK AI Security Institute's Inspect Evals package registers it as `healthbench_hard`, alongside sibling tasks `healthbench` and `healthbench_consensus`, implemented by cloning OpenAI's simple-evals at a pinned commit. OpenAI's own openai/simple-evals repository ships the reference grader in `healthbench_eval.py`, run with a flag selecting the hard subset. Reported scores depend on which grader model is used to apply the rubric and on how conversations are resampled for the bootstrap confidence interval, so cross-paper comparisons are only reliable when the grader model matches.\n\n## Reading the numbers\n\nBecause HealthBench Hard was built to resist saturation, a high score is a meaningfully strong result -- even leading models scored close to 32% at launch. A low score does not necessarily mean a model gives unsafe medical advice; it can also reflect over-caution, excess verbosity, or missing one specific communication behaviour the rubric rewards. Read the Hard score alongside the base HealthBench score and, where available, a per-axis or per-theme breakdown, since one aggregate number can hide whether a model is weak on accuracy, communication or context-handling specifically.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "healthbench_hard",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Muse Spark",
   "model_id": "meta/muse-spark",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 42.8,
   "source": "meta-blog, officechai, artificial-analysis"
  }
 ],
 "page": {
  "aliases": [],
  "category": "domain",
  "contamination": {
   "note": "Conversations and rubrics are fully public, and OpenAI itself re-reports this benchmark across its own model releases, which is a plausible contamination vector over time, but no source consulted states a measured contamination rate for this benchmark.",
   "risk": "unknown"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 1000,
   "size_note": "1,000 conversations: the lowest-average-scoring slice of the full 5,000-conversation HealthBench set, carrying physician-written rubric criteria drawn from HealthBench's 48,000+ total criteria",
   "splits": "single 1,000-example set; no train/test split",
   "url": "https://github.com/openai/simple-evals"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice F",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "healthbench_hard",
   "lm_eval": "",
   "opencompass": "",
   "other": "openai/simple-evals healthbench_eval.py, run with the hard subset flag"
  },
  "id": "healthbench_hard",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "healthbench",
   "successors": [],
   "variants": []
  },
  "measures": "HealthBench Hard tests how a model handles realistic, multi-turn healthcare conversations, standing in for a patient, caregiver or clinician. Each conversation was written or adapted with input from physicians to probe a specific behaviour: giving correct information, communicating clearly, being appropriately cautious, or handling an emergency or context correctly. HealthBench Hard is a fixed 1,000-conversation slice of the full 5,000-conversation HealthBench set, chosen as the conversations that scored lowest, on average, across a panel of model providers when the benchmark was built -- the cases current frontier models handled worst.",
  "metric": {
   "baseline_note": "",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "rubric criteria met (mean score, bootstrap-aggregated)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "HealthBench Hard",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2505.08775",
   "title": "HealthBench: Evaluating Large Language Models Towards Improved Human Health",
   "url": "https://arxiv.org/abs/2505.08775",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Rahul K. Arora",
    "Jason Wei",
    "Rebecca Soskin Hicks",
    "Preston Bowman",
    "Joaquin Quinonero-Candela",
    "Foivos Tsimpourlas",
    "Michael Sharman",
    "Meghan Shah",
    "Andrea Vallone",
    "Alex Beutel",
    "Johannes Heidecke",
    "Karan Singhal"
   ],
   "org": "OpenAI",
   "url": "https://openai.com/index/healthbench/"
  },
  "released": "2025-05",
  "repo_url": "https://github.com/openai/simple-evals",
  "saturation": {
   "as_of": "2025-05",
   "note": "At release the best-performing model reached only about 32% on HealthBench Hard; the subset was built by keeping the lowest-scoring conversations from a multi-provider baseline specifically so it would resist quick saturation. No source consulted here gives a more recent top score.",
   "status": "open",
   "top_score": 32
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "HealthBench: Evaluating Large Language Models Towards Improved Human Health",
    "url": "https://arxiv.org/abs/2505.08775"
   },
   {
    "accessed": "2026-09-07",
    "title": "openai/simple-evals repository",
    "url": "https://github.com/openai/simple-evals"
   },
   {
    "accessed": "2026-09-07",
    "title": "Introducing HealthBench (OpenAI)",
    "url": "https://openai.com/index/healthbench/"
   },
   {
    "accessed": "2026-09-07",
    "title": "inspect_evals healthbench task",
    "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/healthbench"
   }
  ],
  "status": "active",
  "subcategory": "clinical and health conversations, hardest subset",
  "summary": "The 1,000 hardest conversations from OpenAI's HealthBench, graded against physician-written rubrics of what a good health-related response should do.",
  "tags": [
   "health",
   "medicine",
   "rubric-graded",
   "llm-judge",
   "multi-turn"
  ],
  "task_format": "Multi-turn healthcare conversation; the model's final response is graded by an LLM judge against a per-conversation, physician-written rubric of scoring criteria."
 }
}