{
 "body": "\n## What it measures\n\nHumanity's Last Exam tests whether a model can answer closed-ended academic questions at the\nfrontier of expert human knowledge, across dozens of subjects spanning mathematics, the humanities\nand the natural sciences. Each question was written and vetted by a subject-matter expert and\nfiltered so that it cannot be answered quickly by searching the internet, distinguishing it from\nbenchmarks that mostly reward recall or lookup. About 90% of questions are text-only; the rest pair\ntext with a reference image.\n\nThe benchmark was built as a response to earlier knowledge benchmarks like MMLU saturating: frontier\nmodels were clearing 90%+ on MMLU, leaving little room to distinguish further progress. HLE also asks\nmodels to report a numeric confidence alongside each answer, so it measures calibration (whether\nstated confidence tracks actual correctness) as well as raw accuracy.\n\n## How it is scored\n\nModels are graded on accuracy: the percentage of questions answered correctly, judged automatically\nby GPT-4o against the reference answer, using a standardized prompt that structures responses into\n\"Reasoning\" and \"Final Answer\" sections. Because the question set mixes free-text exact-match items\n(about 80%) with multiple-choice items of varying option counts (about 20%), there is no single fixed\nrandom baseline the way a uniform four-option test would have. A second published metric, RMS\ncalibration error, measures the gap between a model's stated confidence and its actual accuracy;\nHLE's authors report this is uniformly poor (above 80% error) across every model they tested at\nlaunch, indicating models answer confidently even when wrong.\n\n## Dataset and licence\n\nHLE launched in January 2025 with 3,000 questions contributed by nearly 1,000 expert reviewers from\nover 500 institutions; later revisions to the paper describe 2,500 questions after removing contested\nor flawed items. The dataset is released under CC BY 4.0 and distributed through Hugging Face\n(`cais/hle`), though access there is gated behind an agreement intended to slow the dataset's entry\ninto future training corpora, alongside an embedded canary string. A private held-out set is kept\nalongside the public release specifically to detect models that have been overfit to the public\nquestions. The authors separately maintain \"HLE-Rolling,\" a continuously refreshed fork that swaps\nin harder questions over time; scores on it are not comparable to the static set.\n\n## Who publishes it\n\nHLE was introduced by the Center for AI Safety (CAIS) and Scale AI, with an organizing team led by\nLong Phan, Alice Gatti, Ziwen Han and Nathaniel Li, and senior authorship from Summer Yue, Alexandr\nWang (Scale AI) and Dan Hendrycks (CAIS). The January 2025 arXiv paper has since been revised\nrepeatedly (through at least a v11 in mid-2026) as the authors refine decontamination and add\nevaluation detail; it was also published in Nature. CAIS runs a public dashboard tracking frontier\nmodel performance on an ongoing basis.\n\n## Lineage\n\nHLE has no formal predecessor; it was created specifically because MMLU and similar knowledge\nbenchmarks had stopped separating frontier models. It shares that motivation with benchmarks like\nGPQA Diamond, though HLE is broader in subject coverage and explicitly multi-modal. This repository\ntracks one direct variant: `hle_tools`, the same question set scored when a model is given tool\naccess (search, browsing, code execution) rather than the untooled default this page describes.\n\n## Saturation and contamination\n\nProgress has been fast relative to the benchmark's age. At the January 2025 release, the strongest\nmodel on the text-only subset (DeepSeek-R1) scored 9.4% against GPT-4o's 3.3%. By November 2025,\nAnthropic's Claude Opus 4.5 System Card measured Gemini 3 at 37.5% without search \u2014 the best of six\nfrontier models it charted \u2014 a roughly four-fold gain in ten months. Scores remain far from any\nceiling, so HLE still separates frontier models well. Contamination risk sits at medium: the public\nquestions carry a canary string and a private held-out set catches overfitting, but the dataset has\ncirculated publicly since January 2025, and Anthropic's November 2025 system card describes actively\nre-grading transcripts that appeared to retrieve answers online during search-enabled runs \u2014 direct\nevidence that browsing access raises leakage risk in practice.\n\n## How to run it\n\nThe authors' repository supplies a reference harness (`run_model_predictions.py` and\n`run_judge_results.py`) built around the OpenAI Python client and a GPT-4o judge, and most published\nnumbers trace back to it. UK AISI's `inspect_evals` package also implements the benchmark as the `hle`\ntask, with options to restrict to the HLE-Verified gold subset or run the rolling variant instead of\nthe static set \u2014 either of which produces scores not comparable to the standard number. Judge choice,\nprompt wording and tool access all affect reported scores, so compare figures across sources with\ncare.\n\n## Reading the numbers\n\nA high HLE score today mostly indicates strong closed-ended recall and reasoning across a broad\nacademic sweep, produced under a prompt that discourages guessing without justification. It does not\nindicate open-ended research capability, and calibration error is worth reading alongside accuracy:\nHLE's results show models are frequently wrong with high stated confidence. Because scores remain\nwell below any plausible ceiling, differences between frontier models here are currently meaningful\nrather than noise. Compare a plain HLE score against `hle_tools` deliberately: the gap shows how much\nof a model's apparent knowledge is retrieval it can do itself versus recall from training.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "hle",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Mythos Preview",
   "model_id": "anthropic/claude-mythos-preview",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 56.8,
   "source": "anthropic-system-card"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Muse Spark",
   "model_id": "meta/muse-spark",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 50.2,
   "source": "meta-blog, officechai, artificial-analysis"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 3.1 Pro Preview",
   "model_id": "google/gemini-3-1-pro-preview",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 44.4,
   "source": "anthropic-system-card-mythos, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.6",
   "model_id": "anthropic/claude-opus-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 40.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, anthropic-system-card-mythos"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.4",
   "model_id": "openai/gpt-5-4",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 39.8,
   "source": "lmarena.ai, provider-reports, anthropic-system-card-mythos, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GLM 5.1",
   "model_id": "zhipu/glm-5-1",
   "provider": "zhipu",
   "provider_display": "Z.ai (Zhipu AI)",
   "score": 31.0,
   "source": "zai-org-model-card"
  }
 ],
 "page": {
  "aliases": [
   "HLE"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "The public release carries a canary string and the authors screened candidate questions against contemporary frontier models before inclusion, but the questions have been public since January 2025 and are a popular target for both training and evaluation. A third-party audit (\"HLE-Verified\") has since re-checked the static set for erroneous or compromised items. Anthropic's Claude Opus 4.5 System Card (November 2025) reports actively flagging and re-grading transcripts in its search-enabled evaluation that had visited known answer-sheet domains or otherwise appeared to retrieve answers rather than derive them, which is direct evidence that contamination is a live risk once a model is allowed to browse.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC BY 4.0",
   "modalities": [
    "text",
    "image"
   ],
   "public_test_set": true,
   "size": 2500,
   "size_note": "The dataset launched in January 2025 with 3,000 questions; the abstract of the arXiv paper, as revised through mid-2026, now describes 2,500, after later revisions removed contested or flawed items. A private held-out set is kept alongside the public set specifically to detect overfitting and gaming. The authors also maintain \"HLE-Rolling,\" a separately versioned dynamic fork that swaps in harder held-out questions over time and is not comparable to the static set.\n",
   "splits": "single public test split, plus a private held-out set not released",
   "url": "https://huggingface.co/datasets/cais/hle"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 1b, slice L",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "hle",
   "lm_eval": "",
   "opencompass": "",
   "other": "The authors' own repository (centerforaisafety/hle) ships run_model_predictions.py and run_judge_results.py, using an automated GPT-4o judge against a standardized reasoning-then-answer prompt; this is the closest thing to an official harness and is what most labs' self-reported numbers are built on.\n"
  },
  "id": "hle",
  "last_updated": "2026-07",
  "leaderboard_url": "https://agi.safe.ai/dashboard",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "hle_tools"
   ]
  },
  "measures": "Humanity's Last Exam (HLE) tests whether a model can answer closed-ended academic questions at the frontier of human expertise, across dozens of subjects including mathematics, the humanities and the natural sciences. Each question was written by a subject-matter expert and is deliberately constructed to have a single, unambiguous, verifiable answer that cannot be produced quickly by searching the internet. About 90% of questions are text-only; the remainder pair text with a reference image, so the benchmark is described by its authors as multi-modal rather than vision-first. The benchmark also scores calibration: whether a model's stated confidence matches how often it is actually correct.\n",
  "metric": {
   "baseline_note": "No fixed random baseline applies because the format mixes exact-match questions (no baseline) with multiple-choice questions of varying option counts. The paper reports a second metric alongside accuracy, RMS calibration error (0-100, lower is better), computed with the method from Hendrycks et al.; HLE does not publish a single human-expert baseline the way some closed-ended benchmarks do, since the questions are selected specifically to stump the domain experts who help write them.\n",
   "direction": "higher_is_better",
   "max_score": 100,
   "name": "accuracy",
   "unit": "%"
  },
  "name": "Humanity's Last Exam",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2501.14249",
   "title": "Humanity's Last Exam",
   "url": "https://arxiv.org/abs/2501.14249",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Long Phan",
    "Alice Gatti",
    "Ziwen Han",
    "Nathaniel Li",
    "Summer Yue",
    "Alexandr Wang",
    "Dan Hendrycks"
   ],
   "org": "Center for AI Safety and Scale AI",
   "url": "https://lastexam.ai"
  },
  "released": "2025-01",
  "repo_url": "https://github.com/centerforaisafety/hle",
  "saturation": {
   "as_of": "2025-11",
   "note": "At the January 2025 release, the best model (DeepSeek-R1, text-only subset) scored 9.4% against GPT-4o's 3.3%. By November 2025, Anthropic's Claude Opus 4.5 System Card measured Gemini 3 at 37.5% without search, the highest of six frontier models it charted (Claude Opus 4.1, Claude Sonnet 4.5, GPT-5, GPT-5 Pro, Claude Opus 4.5, Gemini 3). Scores have risen quickly but remain far below the ceiling, so HLE still separates frontier models by a wide margin.\n",
   "status": "open",
   "top_score": 37.5
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Humanity's Last Exam (arXiv:2501.14249)",
    "url": "https://arxiv.org/abs/2501.14249"
   },
   {
    "accessed": "2026-09-08",
    "title": "Humanity's Last Exam (official site)",
    "url": "https://lastexam.ai"
   },
   {
    "accessed": "2026-09-08",
    "title": "centerforaisafety/hle (README, LICENSE)",
    "url": "https://github.com/centerforaisafety/hle"
   },
   {
    "accessed": "2026-09-08",
    "title": "cais/hle dataset (gated access page)",
    "url": "https://huggingface.co/datasets/cais/hle"
   },
   {
    "accessed": "2026-09-08",
    "title": "System Card: Claude Opus 4.5 (Anthropic, November 2025), Section 2.16",
    "url": "https://www.anthropic.com/claude-opus-4-5-system-card"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals: hle task implementation",
    "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/hle"
   }
  ],
  "status": "active",
  "subcategory": "frontier academic knowledge Q&A",
  "summary": "2,500 expert-written, closed-ended questions spanning dozens of academic subjects, built by CAIS and Scale AI to replace saturated benchmarks like MMLU.",
  "tags": [
   "reasoning",
   "multi-modal",
   "closed-ended",
   "calibration",
   "frontier-knowledge"
  ],
  "task_format": "Single-turn question, either exact-match (a short string or number the model must produce, roughly 80% of items) or multiple-choice with five or more options (the remainder). About 10% of questions include a reference image alongside the text. Models are also asked to state a numeric confidence (0-100%) alongside their answer, which feeds the calibration metric.\n"
 }
}