Humanity's Last Exam

2,500 expert-written, closed-ended questions spanning dozens of academic subjects, built by CAIS and Scale AI to replace saturated benchmarks like MMLU.

Also known as: HLE

unassessed

What it measures

Humanity's Last Exam (HLE) tests whether a model can answer closed-ended academic questions at the frontier of human expertise, across dozens of subjects including mathematics, the humanities and the natural sciences. Each question was written by a subject-matter expert and is deliberately constructed to have a single, unambiguous, verifiable answer that cannot be produced quickly by searching the internet. About 90% of questions are text-only; the remainder pair text with a reference image, so the benchmark is described by its authors as multi-modal rather than vision-first. The benchmark also scores calibration: whether a model's stated confidence matches how often it is actually correct.

Task format

Single-turn question, either exact-match (a short string or number the model must produce, roughly 80% of items) or multiple-choice with five or more options (the remainder). About 10% of questions include a reference image alongside the text. Models are also asked to state a numeric confidence (0-100%) alongside their answer, which feeds the calibration metric.

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub