{
 "body": "\n## What it measures\n\nArena Elo measures which chatbot response people prefer in head-to-head, blind comparisons \u2014\na live measurement of human preference on real, unscripted prompts, not accuracy against a\nfixed answer key. A person submits a prompt, receives responses from two anonymised models\nchosen at random, and votes for the one they prefer; model identities are revealed only after\nvoting. Prompts come from the platform's own users rather than a curated test set, so the mix\nof topics and difficulty shifts over time and by audience. This repository's subset pages\ncover the overall text leaderboard plus five commonly quoted slices: coding, math, a curated\n\"hard prompts\" subset, a style-controlled ranking, and a separate vision (image+text)\nleaderboard.\n\n## How it is scored\n\nEach vote is a pairwise outcome (A wins, B wins, or tie) between two models. The platform\noriginally computed ratings with the classic sequential Elo update (the same logistic,\nbase-10 formula used in chess), but has since moved to a Bradley-Terry model fit by maximum\nlikelihood over the complete vote history, which treats model strength as fixed and produces\nmaterially more stable ratings with real 95% confidence intervals, rather than ratings that\ndepend on the order votes happened to arrive in. Ratings sit on an anchored scale with no\nfixed maximum, so unlike an accuracy metric there is no \"100%\" to reach. The Style Control\nvariant refits the same model after regressing out response length and markdown-formatting\neffects, to separate substance from presentation.\n\n## Dataset and licence\n\nThere is no static downloadable dataset the way FLORES or MTEB have one: the \"dataset\" is the\nlive, ever-growing corpus of anonymous pairwise votes, which stood above 240,000 at the time\nof the 2024 paper and has grown enormously since. The organisation periodically releases\nhistorical snapshots of vote and conversation data as open datasets on Hugging Face\n(lmarena-ai). A specific licence for the live voting data was not established from a source\nread for this page; consult the platform's own Terms of Use and Leaderboard Policy pages for\ncurrent terms.\n\n## Who publishes it\n\nChatbot Arena was launched in April 2023 by LMSYS (Large Model Systems Organization), a group\ncentred at UC Berkeley's Sky Computing Lab, and first written up in a May 2023 blog post by\nLianmin Zheng, Ying Sheng, Wei-Lin Chiang, Hao Zhang, Joseph E. Gonzalez and Ion Stoica. The\nformal paper, \"Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference\" (arXiv,\nMarch 2024; ICML 2024), adds Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua\nZhu and Michael Jordan to the author list. The project later spun out as its own company,\nrebranding first to LMArena and, by 2026, operating as Arena at arena.ai.\n\n## Lineage\n\nArena Elo has no single predecessor benchmark; its founders cite the general-purpose Elo\nrating system from chess and note that an earlier Anthropic paper had already applied\nElo-style rating to language models. In this repository, `arena_elo_overall` is the headline\ntext leaderboard, and `arena_elo_coding`, `arena_elo_math`, `arena_elo_hard_prompts`,\n`arena_elo_style_control` and `arena_elo_vision` are its category and modality subsets. The\nplatform has since spawned related, separately named efforts from the same organisation \u2014\nArena-Hard-Auto (an offline, judge-based proxy), RouteLLM, Search Arena, and dedicated WebDev,\nDocument and Text-to-Image arenas \u2014 none of which are in scope for this family page.\n\n## Saturation and contamination\n\nRatings are live and continuously updated as new models and votes arrive, and frontier models\nremain within a tight but statistically real band given their reported confidence intervals,\nrather than piled up at a hard ceiling \u2014 there is no fixed maximum on an Elo-style scale, so\n\"saturation\" in the sense a fixed-answer benchmark uses the word does not quite apply.\nContamination in the train/test-leakage sense is low, since prompts are not a fixed,\npublishable answer key a model could be trained on in advance. The live-voting format brings a\ndifferent validity risk instead: the organisation has published its own research on\nadversarial gaming of leaderboards and on how private, unreleased models are probed anonymously\nin the arena before launch, both of which affect how much to trust a ranking rather than\nwhether the test set leaked.\n\n## How to run it\n\nThere is no offline harness: ratings come only from live human votes on the platform,\ncollected through its own web interface (originally served by the open-source FastChat\nsystem). A team cannot \"run\" Arena Elo against its own model outside the live platform, though\nthe organisation's periodic open vote-data releases and its Arena-Hard-Auto tool let\nresearchers approximate arena-style preference judging offline, without new live human votes.\nNumbers are hard to compare across snapshots because the pool of competing models, the prompt\nmix, and the rating method itself (sequential Elo before 2024, Bradley-Terry since) have all\nchanged over the platform's history.\n\n## Reading the numbers\n\nA model's Arena Elo score reflects what a broad, self-selected population of platform users\npreferred on the prompts they happened to submit, filtered through whichever category\n(coding, math, hard prompts) and style-control setting you are looking at \u2014 it is not an\naccuracy score and does not verify correctness, only preference. Always check the reported\nconfidence interval before treating a small ranking gap as meaningful: intervals of several\npoints or more are common, and two overlapping intervals mean the ranking between those models\nis not statistically settled. Prefer the style-controlled and hard-prompts views over the raw\noverall score if you specifically want to filter out response-length and formatting bias.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "arena_elo",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Chatbot Arena",
   "LMArena",
   "Arena"
  ],
  "category": "human-preference",
  "contamination": {
   "note": "Prompts are live user submissions, not a fixed, publishable answer key a model could be trained on in advance. The organisation has instead published its own research on adversarial gaming of leaderboards and on how unreleased models are probed anonymously before launch \u2014 a different validity risk from classic train/test leakage.",
   "risk": "low"
  },
  "dataset": {
   "languages": [],
   "license": "",
   "modalities": [
    "text",
    "image"
   ],
   "public_test_set": null,
   "size": null,
   "size_note": "No static dataset: a live, ever-growing corpus of anonymous pairwise votes, above 240,000 at the time of the March 2024 paper and far larger since. Historical snapshots are released periodically as open datasets on Hugging Face (lmarena-ai).",
   "splits": "",
   "url": "https://huggingface.co/lmarena-ai"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice D",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "No offline harness: ratings come only from live human votes on the platform (originally served by the open-source FastChat system). The organisation's periodic open vote-data releases and its Arena-Hard-Auto tool approximate arena-style judging offline."
  },
  "id": "arena_elo",
  "last_updated": "",
  "leaderboard_url": "https://lmarena.ai/leaderboard",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "arena_elo_overall",
    "arena_elo_coding",
    "arena_elo_math",
    "arena_elo_hard_prompts",
    "arena_elo_style_control",
    "arena_elo_vision"
   ]
  },
  "measures": "Arena Elo measures which chatbot response people prefer in head-to-head, blind comparisons \u2014 a live measurement of human preference on real, unscripted prompts, not accuracy against a fixed answer key. A person submits a prompt, receives responses from two anonymised models chosen at random, and votes for the one they prefer; model identities are revealed only after voting. This repository's subsets cover the overall text leaderboard plus five commonly quoted slices: coding, math, a curated \"hard prompts\" subset, a style-controlled ranking, and a separate vision (image+text) leaderboard.\n",
  "metric": {
   "baseline_note": "No fixed random baseline or maximum: ratings are relative to the pool of models currently rated, on an anchored but open-ended scale.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "Elo",
   "random_baseline": null,
   "unit": ""
  },
  "name": "Arena Elo (Chatbot Arena / LMArena)",
  "page_kind": "family",
  "paper": {
   "arxiv": "2403.04132",
   "title": "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference",
   "url": "https://arxiv.org/abs/2403.04132",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Wei-Lin Chiang",
    "Lianmin Zheng",
    "Ying Sheng",
    "Anastasios Nikolas Angelopoulos",
    "Tianle Li",
    "Dacheng Li",
    "Hao Zhang",
    "Banghua Zhu",
    "Michael Jordan",
    "Joseph E. Gonzalez",
    "Ion Stoica"
   ],
   "org": "LMSYS (Large Model Systems Organization), UC Berkeley Sky Computing Lab (founding org); operates today as Arena (formerly LMArena)",
   "url": "https://arena.ai"
  },
  "released": "2023-04",
  "repo_url": "https://github.com/lm-sys/FastChat",
  "saturation": {
   "as_of": "",
   "note": "Ratings are live and continuously updated as new models and votes arrive; frontier models remain within a tight but statistically real band given reported confidence intervals, rather than piled up at a hard ceiling, since an Elo-style scale has no fixed maximum.",
   "status": "open",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (arXiv)",
    "url": "https://arxiv.org/abs/2403.04132"
   },
   {
    "accessed": "2026-09-07",
    "title": "Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings (LMSYS blog, founding post)",
    "url": "https://www.lmsys.org/blog/2023-05-03-arena/"
   },
   {
    "accessed": "2026-09-07",
    "title": "How It Works (Arena)",
    "url": "https://arena.ai/how-it-works"
   },
   {
    "accessed": "2026-09-07",
    "title": "Arena Leaderboard Overview",
    "url": "https://lmarena.ai/leaderboard"
   }
  ],
  "status": "active",
  "subcategory": "pairwise human preference",
  "summary": "Elo-style ratings of chat models derived from live, anonymous, pairwise human-preference votes on real user prompts.",
  "tags": [
   "human-preference",
   "chatbot",
   "elo",
   "live-leaderboard"
  ],
  "task_format": "Anonymous, randomized side-by-side chat: a user prompts two randomly paired models and votes for the preferred response, or ties/both-bad."
 }
}