{
 "body": "\n## What it measures\n\nLiveBench evaluates a model across six skill categories in one suite: math, coding, reasoning, data\nanalysis, language comprehension and instruction following. Each category bundles several distinct task\ntypes rather than one narrow format \u2014 competition-math problems, LeetCode/AtCoder-style code generation,\nZebra-puzzle and Web-of-Lies-style logic tasks, table reformatting and column-type annotation, and\nparaphrase/summarize/simplify instruction tasks among them \u2014 so a category score is itself a small suite\naverage, not one task repeated. The live public leaderboard now also tracks a seventh category, Agentic\nCoding, added to the site after the original paper.\n\nThe point of the design is contamination resistance rather than task novelty: questions are built from\nmaterial recent enough that a model could not have trained on the specific item, and every answer is\nchecked automatically rather than scored by another model acting as judge.\n\n## How it is scored\n\nEach task carries a verifiable, objective ground-truth answer \u2014 a number, a passing test suite, a puzzle\nsolution, a correctly reformatted table \u2014 so grading is automatic rather than routed through an LLM\njudge, which the authors designed specifically to avoid the judge-bias and stylistic-preference problems\nof chat-arena-style benchmarks. Category scores and an overall average are reported on a 0\u2013100 scale. No\nsingle random-guess or human baseline applies across such heterogeneous task types, and none was found\nstated as one number by the authors.\n\n## Dataset and licence\n\nAt the original June 2024 release the suite held about 1,000 questions across 18 tasks in the six\noriginal categories. The Hugging Face dataset repositories (one per category, under the `livebench` org)\naccumulate every monthly release rather than overwriting it, tagging each row with\n`livebench_release_date` and `livebench_removal_date` columns; at their last update (7 April 2025) the\nsix repos totalled 1,436 rows, most already marked retired. No `license` metadata is set on any of those\nsix dataset repos, and the GitHub repository's own LICENSE file does not state a single top-level licence\nfor LiveBench's new questions \u2014 it instead bundles an Apache-2.0 notice inherited from reused FastChat\ncode and an MIT notice from reused LiveCodeBench code. This page leaves the licence field blank rather\nthan guess. Text only, English, sourced from recent math competitions, arXiv papers, news articles, IMDb\nsynopses and remixed existing datasets.\n\n## Who publishes it\n\nLiveBench comes from an 18-author academic collaboration \u2014 Colin White, Samuel Dooley, Manley Roberts and\n15 co-authors including Yann LeCun, Tom Goldstein and Chinmay Hegde \u2014 posted to arXiv in June 2024 and\naccepted as a Spotlight paper at ICLR 2025. The team continues to maintain the leaderboard and dataset at\nlivebench.ai and github.com/livebench/livebench, and keeps publishing new monthly question releases well\npast the original paper.\n\n## Lineage\n\nLiveBench names no single predecessor. Its design responds explicitly to weaknesses the authors identify\nin two earlier evaluation styles: static multiple-choice benchmarks, which degrade as they age into\ntraining data, and LLM-judge chat benchmarks such as MT-Bench and Chatbot Arena, which are subject to\njudge-model bias. It has no official successor and no variant pages in this repository.\n\n## Saturation and contamination\n\nThe public leaderboard, as accessed for this page, showed a top overall score of 83.4 (out of 100) with\ncategory scores spread from the high 60s to high 90s \u2014 well short of a ceiling, so the benchmark still\nseparates models. But the leaderboard view itself does not label which monthly release that snapshot\nreflects, which is the central caveat for reading any LiveBench number (see below). Contamination risk is\nlow by explicit design: roughly a sixth of the question pool rotates monthly, retirement prioritises the\noldest and currently-easiest items, a full refresh is targeted about every six months, and each new batch\nis held private for a further month before publication so the live leaderboard always includes at least\none round of questions no released model could have trained on.\n\n## How to run it\n\ninspect_evals registers the task as `inspect_evals/livebench`, with a `category` parameter to select one\nof the six original categories and a `livebench_release_date` parameter (YYYY-MM-DD) to pin a run to one\nspecific monthly question set rather than whichever is current. The reference implementation lives in the\nauthors' own GitHub repository, which reuses evaluation and serving infrastructure from FastChat and\ngrading code from LiveCodeBench. No lm-evaluation-harness, HELM, OpenCompass or BIG-bench task was\nconfirmed for LiveBench in this research.\n\n## Reading the numbers\n\nAlways check which monthly release a LiveBench number was measured against before comparing it with\nanother one: because roughly a sixth of questions rotate every month and old ones are retired, two scores\nfor the same model taken months apart are not guaranteed to share a single question. Pin the release with\n`livebench_release_date` when reproducibility matters. Because grading is automatic against objective\nground truth, LiveBench avoids the judge-model bias that affects arena-style chat benchmarks, but that\nsame design confines it to task types with a clear right answer, so it says nothing about open-ended\ndialogue quality. Read a LiveBench score alongside its release date and category breakdown, not as a\nsingle static number.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "livebench",
  "reasons": [
   "no qualifying current frontier/open coverage from different organizations",
   "review is not approved",
   "task, metric, and protocol are incomplete",
   "usefulness is not established",
   "usefulness is unknown"
  ],
  "status": "unverified",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Live Bench"
  ],
  "category": "composite",
  "contamination": {
   "note": "Contamination risk is low by explicit design rather than by luck. LiveBench sources questions from recently published material (recent competitions, arXiv papers, news articles, IMDb synopses, and remixed existing datasets), releases roughly one-sixth of its question pool as new each month, prioritises retiring the oldest and currently-easiest items, and targets a full refresh about every six months. Each month's newly written batch is kept private for a further month before publication, so the live leaderboard always includes at least one round of not-yet-public questions no model could have trained on.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": null,
   "size": 1436,
   "size_note": "At the original June 2024 paper release: about 1,000 questions across 18 tasks in the 6 original categories. The Hugging Face dataset repos accumulate every monthly release rather than overwriting it, tagging each row with livebench_release_date and livebench_removal_date columns; by their last update (7 April 2025) the six per-category repos totalled 1,436 rows (128 coding, 368 math, 200 reasoning, 190 language, 150 data analysis, 400 instruction-following), most of them since retired. This is a cumulative historical count, not the size of any single month's active question set.\n",
   "splits": "no fixed train/test split; questions are released monthly per category and later marked retired via a removal-date field rather than deleted",
   "url": "https://huggingface.co/livebench"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 2, slice A",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "livebench",
   "lm_eval": "",
   "opencompass": "",
   "other": "inspect_evals/livebench takes a `category` parameter (math, reasoning, coding, language, data_analysis, instruction_following) and a `livebench_release_date` parameter (YYYY-MM-DD) that pins a run to one specific monthly question set instead of whichever is current. No lm-evaluation-harness, HELM, OpenCompass or BIG-bench task was confirmed for LiveBench in this research.\n"
  },
  "id": "livebench",
  "last_updated": "",
  "leaderboard_url": "https://livebench.ai/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "LiveBench evaluates a model across six skill categories in one suite: math, coding, reasoning, data analysis, language comprehension and instruction following. Each category bundles several distinct task types rather than one narrow format \u2014 for example competition-math problems, LeetCode/AtCoder-style code generation, Zebra-puzzle and Web-of-Lies-style logic tasks, table reformatting and column-type annotation, and paraphrase/summarize/simplify instruction tasks \u2014 so a category score is itself a small suite average. The public leaderboard now also tracks a seventh category, Agentic Coding, added after the original paper.\n",
  "metric": {
   "baseline_note": "Tasks are heterogeneous enough (free-form code, puzzle answers, reformatted tables) that no single random-guess or human baseline applies across the whole suite; neither was found stated as one number by the authors.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "task-specific automatic accuracy, averaged per category and overall",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "LiveBench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2406.19314",
   "title": "LiveBench: A Challenging, Contamination-Free LLM Benchmark",
   "url": "https://arxiv.org/abs/2406.19314",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Colin White",
    "Samuel Dooley",
    "Manley Roberts",
    "Arka Pal",
    "Ben Feuer",
    "Siddhartha Jain",
    "Ravid Shwartz-Ziv",
    "Neel Jain",
    "Khalid Saifullah",
    "Sreemanti Dey",
    "Shubh Agrawal",
    "Sandeep Singh Sandha",
    "Siddhartha Naidu",
    "Chinmay Hegde",
    "Yann LeCun",
    "Tom Goldstein",
    "Willie Neiswanger",
    "Micah Goldblum"
   ],
   "org": "",
   "url": "https://livebench.ai"
  },
  "released": "2024-06",
  "repo_url": "https://github.com/LiveBench/LiveBench",
  "saturation": {
   "as_of": "2026-09",
   "note": "83.4 overall (Claude Fable 5.1 Max Effort) was the top score on the public livebench.ai leaderboard as accessed on 2026-09-08. The leaderboard view itself does not label which monthly question-set release that snapshot draws on, which is exactly the ambiguity LiveBench's own design makes possible \u2014 see \"Reading the numbers.\" Category spreads on that same snapshot ranged from the high 60s to high 90s, well short of a ceiling, so the benchmark still separates models.\n",
   "status": "open",
   "top_score": 83.4
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "LiveBench: A Challenging, Contamination-Free LLM Benchmark",
    "url": "https://arxiv.org/abs/2406.19314"
   },
   {
    "accessed": "2026-09-08",
    "title": "LiveBench/LiveBench repository (README, LICENSE)",
    "url": "https://github.com/LiveBench/LiveBench"
   },
   {
    "accessed": "2026-09-08",
    "title": "livebench/coding dataset card and structure",
    "url": "https://huggingface.co/datasets/livebench/coding"
   },
   {
    "accessed": "2026-09-08",
    "title": "livebench/math dataset structure",
    "url": "https://huggingface.co/datasets/livebench/math"
   },
   {
    "accessed": "2026-09-08",
    "title": "livebench/instruction_following dataset structure",
    "url": "https://huggingface.co/datasets/livebench/instruction_following"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals livebench task (category and livebench_release_date parameters)",
    "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/livebench"
   },
   {
    "accessed": "2026-09-08",
    "title": "LiveBench.AI leaderboard (fetched via scripts/benchmarks/fetch.py)",
    "url": "https://livebench.ai/"
   }
  ],
  "status": "active",
  "subcategory": "monthly-refreshed multi-category benchmark",
  "summary": "A monthly-refreshed suite across math, coding, reasoning, data analysis, language and instruction following, graded by automatic ground truth rather than an LLM judge, to limit contamination.",
  "tags": [
   "contamination-resistant",
   "multi-category",
   "monthly-refresh",
   "composite",
   "math",
   "coding",
   "reasoning",
   "objective-grading"
  ],
  "task_format": "Mostly single-turn text prompts (a small number of tasks use multiple turns) with a task-specific expected output \u2014 a number, a code solution graded by test cases, a puzzle answer, a reformatted table \u2014 that is checked automatically rather than judged by another model.\n"
 }
}