{
 "body": "\n## What it measures\n\nMATH-500 gives a model 500 written competition mathematics problems drawn from algebra, geometry,\nnumber theory, counting and probability, precalculus, intermediate algebra and prealgebra. The\nmodel is given a problem statement in English and must produce a worked solution and a final\nanswer as free text; there are no answer choices to pick from. It exercises multi-step symbolic\nand numeric reasoning: understanding what a problem is asking, planning a solution route, and\nexecuting algebraic or geometric manipulation correctly. It does not test speed, tool use, or any\nlanguage beyond English.\n\n## How it is scored\n\nGrading compares a model's final answer against the reference answer, not the reasoning that\nproduced it. OpenAI's reference implementation (simple-evals, `math_eval.py`) asks for the final\nline of a response to read \"Answer: $ANSWER\", extracts that line with a regular expression, and\nchecks it against the reference using a separate LLM call as an equality checker, since a correct\nanswer can be written more than one way (1/2 vs 0.5, for example). Reported numbers are usually\npass@1 accuracy under this protocol, but papers differ on sampling temperature, number of samples,\nand whether they take a majority vote over several completions, so two MATH-500 numbers are only\ndirectly comparable once you know both used the same protocol. No confirmed random-guess or human\nbaseline exists for this specific 500-item split.\n\n## Dataset and licence\n\nMATH-500 is 500 problems held out from the 5,000-problem test split of the original MATH dataset\n(Hendrycks et al., 2021). OpenAI moved the other 4,500 MATH test problems into their training set\nfor process-supervision work and evaluated only on these 500, describing them as a representative,\nindependent and identically distributed sample of the original test split. Each item has a problem\nstatement, a full worked solution, a final answer, a subject label (one of seven categories such as\nalgebra, geometry or number theory) and a difficulty level from 1 to 5. It ships as a single \"test\"\nsplit of 500 rows of English text, and the answers are public: there is no held-out grading server.\nNo licence is stated on the HuggingFaceH4/MATH-500 dataset card. The source corpus is legally\ncontested: Art of Problem Solving filed a DMCA takedown against the `hendrycks/competition_math`\ndataset in January 2025, stating that more than 10,000 of the 12,500 MATH problems were copied from\nits Alcumus learning platform, and asked that downstream datasets derived from MATH be removed too.\nMATH-500 remains hosted on Hugging Face as of this writing, but its licensing status is unresolved.\n\n## Who publishes it\n\nThe 500-problem split was defined by Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards,\nBowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever and Karl Cobbe at OpenAI, in \"Let's\nVerify Step by Step\" (arXiv 2305.20050, posted May 2023, published at ICLR 2024). It builds on the\nMATH dataset published by Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart,\nEric Tang, Dawn Song and Jacob Steinhardt at NeurIPS 2021. OpenAI's own simple-evals repository uses\nMATH-500 as its default math evaluation for o1-class models onward, and independent trackers such as\nArtificial Analysis run their own copy of the eval and publish current scores.\n\n## Lineage\n\nMATH-500 is a fixed subset of the MATH test set (Hendrycks et al., 2021); the full 12,500-problem\nMATH dataset does not yet have its own page in this repository. MATH-500 has no announced successor\nof its own. The same contamination pressure that pushed OpenAI toward a smaller held-out set has\nsince pushed newer math evaluations toward problems dated after a model's training cutoff, a route\nMATH-500 cannot take, since its 500 problems and reference answers have been public since 2023.\n\n## Saturation and contamination\n\nMATH-500 is saturated for frontier models. On Artificial Analysis's independently run leaderboard\n(accessed September 2026), GPT-5 (high) scores 99.4%, with o3, Grok 3 mini Reasoning (high), GPT-5\n(medium) and Claude 4 Sonnet all at 99.1-99.2%; the top of the leaderboard is separated by tenths of\na point. Contamination risk is high: the answers are distributed in the open dataset, the source\nMATH corpus has circulated since 2021, and Wu et al. (arXiv 2507.10532, 2025) report\nmemorization-consistent behaviour by Qwen2.5-series models specifically on MATH-500, which casts\ndoubt on reinforcement-learning gains measured against it. A high score today separates strong\nmodels from each other by very little, and says nothing about whether that separation is genuine.\n\n## How to run it\n\nOpenAI's openai/simple-evals repository is the reference implementation (`math_eval.py`), which\nloads the `math_500_test` split and grades with the equality-checker protocol described above. The\ndataset is also mirrored at HuggingFaceH4/MATH-500. lm-evaluation-harness ships two tasks for this\nsplit, `hendrycks_math500` and `minerva_math500`, both loading `HuggingFaceH4/MATH-500` directly and\ndiffering only in the answer-extraction and grading style they inherit from `hendrycks_math` and\n`minerva_math` respectively; those two parent tasks still run the full 5,000-item set, so check which\ntask name produced a number before comparing it. Because grading depends on an LLM\nequality checker rather than exact string match, scores depend on which model performs that check\nand on the exact prompt template, a common source of small cross-paper differences.\n\n## Reading the numbers\n\nA high MATH-500 score mainly shows that a model can execute multi-step symbolic manipulation\nreliably and format its answer the way the grader expects. It does not show the model is free of\ncontamination on this exact test set; given the DMCA dispute over the source data and the\nmemorization findings above, treat any single MATH-500 number with more caution than a freshly\nbuilt benchmark deserves. Because the ceiling is close to 100%, small differences between top models\nare not meaningful on their own; a harder, more recent competition-math benchmark separates frontier\nmodels better. Before comparing two reported MATH-500 numbers, check the sampling and grading\nprotocol behind each one.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "math_500",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 0528",
   "model_id": "deepseek/deepseek-r1-0528",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 97.5,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 0528 NVFP4 v2",
   "model_id": "nvidia/deepseek-r1-0528-nvfp4-v2",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 97.5,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1",
   "model_id": "deepseek/deepseek-r1",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 97.3,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek Reasoner",
   "model_id": "deepseek/deepseek-reasoner",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 97.3,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o3-pro",
   "model_id": "openai/o3-pro",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 97.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.1",
   "model_id": "openai/gpt-5-1",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 96.8,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.1 Chat",
   "model_id": "openai/gpt-5-1-chat-latest",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 96.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.1 Codex",
   "model_id": "openai/gpt-5-1-codex",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 96.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.1 Codex Max",
   "model_id": "openai/gpt-5-1-codex-max",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 96.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.1 Codex mini",
   "model_id": "openai/gpt-5-1-codex-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 96.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o3",
   "model_id": "openai/o3",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 96.7,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o3-deep-research",
   "model_id": "openai/o3-deep-research",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 96.7,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Mythos Preview",
   "model_id": "anthropic/claude-mythos-preview",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 96.4,
   "source": "anthropic-system-card"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4",
   "model_id": "anthropic/claude-opus-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 96.4,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.6",
   "model_id": "anthropic/claude-opus-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 96.4,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, anthropic-system-card-mythos"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o4-mini",
   "model_id": "openai/o4-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.8,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o4-mini-deep-research",
   "model_id": "openai/o4-mini-deep-research",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o1-pro",
   "model_id": "openai/o1-pro",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.6,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5",
   "model_id": "openai/gpt-5",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.5,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5 Chat (latest)",
   "model_id": "openai/gpt-5-chat-latest",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5 Pro",
   "model_id": "openai/gpt-5-pro",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5-Codex",
   "model_id": "openai/gpt-5-codex",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.2",
   "model_id": "openai/gpt-5-2",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.5,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.2 Chat",
   "model_id": "openai/gpt-5-2-chat-latest",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.2 Codex",
   "model_id": "openai/gpt-5-2-codex",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.2 Pro",
   "model_id": "openai/gpt-5-2-pro",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.3 Chat (latest)",
   "model_id": "openai/gpt-5-3-chat-latest",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.3 Codex",
   "model_id": "openai/gpt-5-3-codex",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.3 Codex Spark",
   "model_id": "openai/gpt-5-3-codex-spark",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.4",
   "model_id": "openai/gpt-5-4",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.5,
   "source": "lmarena.ai, provider-reports, anthropic-system-card-mythos, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.4 mini",
   "model_id": "openai/gpt-5-4-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.4 nano",
   "model_id": "openai/gpt-5-4-nano",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.4 Pro",
   "model_id": "openai/gpt-5-4-pro",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.5",
   "model_id": "anthropic/claude-opus-4-5-20251101",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 95.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.5 (latest)",
   "model_id": "anthropic/claude-opus-4-5",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 95.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro",
   "model_id": "google/gemini-2-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 95.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 05-06",
   "model_id": "google/gemini-2-5-pro-preview-05-06",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 95.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 06-05",
   "model_id": "google/gemini-2-5-pro-preview-06-05",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 95.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview TTS",
   "model_id": "google/gemini-2-5-pro-preview-tts",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 95.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o3-mini",
   "model_id": "openai/o3-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 94.8,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.1",
   "model_id": "anthropic/claude-opus-4-1-20250805",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 94.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.1 (latest)",
   "model_id": "anthropic/claude-opus-4-1",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 94.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4",
   "model_id": "xai/grok-4",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 94.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4 Fast",
   "model_id": "xai/grok-4-fast",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 94.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4 Fast (Non-Reasoning)",
   "model_id": "xai/grok-4-fast-non-reasoning",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 94.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4.1 Fast",
   "model_id": "xai/grok-4-1-fast",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 94.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4.1 Fast (Non-Reasoning)",
   "model_id": "xai/grok-4-1-fast-non-reasoning",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 94.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4.20 (Non-Reasoning)",
   "model_id": "xai/grok-4-20-0309-non-reasoning",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 94.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4.20 (Reasoning)",
   "model_id": "xai/grok-4-20-0309-reasoning",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 94.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4.20 Multi-Agent",
   "model_id": "xai/grok-4-20-multi-agent-0309",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 94.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4",
   "model_id": "anthropic/claude-sonnet-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 93.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.6",
   "model_id": "anthropic/claude-sonnet-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 93.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 235B-A22B",
   "model_id": "qwen/qwen3-235b-a22b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 93.5,
   "source": "lmarena.ai, provider-reports, preference-evals, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4 (latest)",
   "model_id": "anthropic/claude-opus-4-0",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 92.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3.2",
   "model_id": "deepseek/deepseek-v3-2",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 92.5,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3.2 Exp",
   "model_id": "deepseek/deepseek-v3-2-exp",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 92.5,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o1",
   "model_id": "openai/o1",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 92.5,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o1-preview",
   "model_id": "openai/o1-preview",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 92.5,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.5",
   "model_id": "anthropic/claude-sonnet-4-5-20250929",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 92.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.5 (latest)",
   "model_id": "anthropic/claude-sonnet-4-5",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 92.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3-Coder 480B-A35B Instruct",
   "model_id": "qwen/qwen3-coder-480b-a35b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 91.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3.1",
   "model_id": "deepseek/deepseek-v3-1",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 91.0,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash",
   "model_id": "google/gemini-2-5-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 90.5,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Image",
   "model_id": "google/gemini-2-5-flash-image",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 90.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Image (Preview)",
   "model_id": "google/gemini-2-5-flash-image-preview",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 90.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Lite",
   "model_id": "google/gemini-2-5-flash-lite",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 90.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Lite Preview 06-17",
   "model_id": "google/gemini-2-5-flash-lite-preview-06-17",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 90.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Lite Preview 09-25",
   "model_id": "google/gemini-2-5-flash-lite-preview-09-2025",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 90.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Preview 04-17",
   "model_id": "google/gemini-2-5-flash-preview-04-17",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 90.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Preview 05-20",
   "model_id": "google/gemini-2-5-flash-preview-05-20",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 90.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Preview 09-25",
   "model_id": "google/gemini-2-5-flash-preview-09-2025",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 90.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Preview TTS",
   "model_id": "google/gemini-2-5-flash-preview-tts",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 90.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "QwQ Plus",
   "model_id": "qwen/qwq-plus",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 90.2,
   "source": "provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o1-mini",
   "model_id": "openai/o1-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 90.0,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1",
   "model_id": "openai/gpt-4-1",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 89.4,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, domain-evals preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek Chat",
   "model_id": "deepseek/deepseek-chat",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 89.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V2",
   "model_id": "deepseek/deepseek-v2",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 89.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V2 Lite",
   "model_id": "deepseek/deepseek-v2-lite",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 89.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V2 Lite Chat",
   "model_id": "deepseek/deepseek-v2-lite-chat",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 89.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3",
   "model_id": "deepseek/deepseek-v3",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 89.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, domain-evals open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3 0324",
   "model_id": "deepseek/deepseek-v3-0324",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 89.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5 Mini",
   "model_id": "openai/gpt-5-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 89.1,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4 (latest)",
   "model_id": "anthropic/claude-sonnet-4-0",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 88.6,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Llama 70B",
   "model_id": "deepseek/deepseek-r1-distill-llama-70b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 88.5,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 32B",
   "model_id": "qwen/qwen3-32b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 88.5,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 32B AWQ",
   "model_id": "qwen/qwen3-32b-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 88.5,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 32B NVFP4",
   "model_id": "nvidia/qwen3-32b-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 88.5,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 31B",
   "model_id": "google/gemma-4-31b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 31B it",
   "model_id": "google/gemma-4-31b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 31B it GGUF",
   "model_id": "unsloth/gemma-4-31b-it-gguf",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 87.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 31B IT NVFP4",
   "model_id": "nvidia/gemma-4-31b-it-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 87.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Qwen 32B",
   "model_id": "deepseek/deepseek-r1-distill-qwen-32b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 86.2,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.7",
   "model_id": "anthropic/claude-3-7-sonnet-20250219",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 85.5,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Magistral Medium (latest)",
   "model_id": "mistral/magistral-medium-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 85.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 26B",
   "model_id": "google/gemma-4-26b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 85.1,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 26B A4B it",
   "model_id": "google/gemma-4-26b-a4b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 85.1,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 26B A4B it GGUF",
   "model_id": "unsloth/gemma-4-26b-a4b-it-gguf",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 85.1,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 4.5",
   "model_id": "anthropic/claude-haiku-4-5-20251001",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 84.3,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 4.5 (latest)",
   "model_id": "anthropic/claude-haiku-4-5",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 84.3,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 30B A3B Instruct 2507",
   "model_id": "qwen/qwen3-30b-a3b-instruct-2507",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 84.2,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 30B A3B NVFP4",
   "model_id": "nvidia/qwen3-30b-a3b-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 84.2,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 30B-A3B",
   "model_id": "qwen/qwen3-30b-a3b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 84.2,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 72B Instruct",
   "model_id": "qwen/qwen2-5-72b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 83.2,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3",
   "model_id": "xai/grok-3",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3 Fast",
   "model_id": "xai/grok-3-fast",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3 Fast Latest",
   "model_id": "xai/grok-3-fast-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3 Latest",
   "model_id": "xai/grok-3-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "phi 4",
   "model_id": "microsoft/phi-4",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2,, domain-evals llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Phi 4 multimodal instruct",
   "model_id": "microsoft/phi-4-multimodal-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.0 Flash",
   "model_id": "google/gemini-2-0-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 82.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1 mini",
   "model_id": "openai/gpt-4-1-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 82.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Maverick 17B 128E Instruct",
   "model_id": "meta/llama-4-maverick-17b-128e-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 82.2,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-4-Maverick-17B-128E-Instruct-FP8",
   "model_id": "meta/llama-4-maverick-17b-128e-instruct-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 82.2,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Qwen 14B",
   "model_id": "deepseek/deepseek-r1-distill-qwen-14b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 82.1,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3 Mini",
   "model_id": "xai/grok-3-mini",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 80.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3 Mini Fast",
   "model_id": "xai/grok-3-mini-fast",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 80.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3 Mini Fast Latest",
   "model_id": "xai/grok-3-mini-fast-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 80.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3 Mini Latest",
   "model_id": "xai/grok-3-mini-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 80.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 14B",
   "model_id": "qwen/qwen3-14b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 80.5,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 14B AWQ",
   "model_id": "qwen/qwen3-14b-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 80.5,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 14B NVFP4",
   "model_id": "nvidia/qwen3-14b-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 80.5,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large (latest)",
   "model_id": "mistral/mistral-large-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 78.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large 2.1",
   "model_id": "mistral/mistral-large-2411",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 78.5,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals,, domain-evals open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large 3",
   "model_id": "mistral/mistral-large-2512",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 78.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5",
   "model_id": "anthropic/claude-3-5-sonnet-20240620",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 78.3,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5 v2",
   "model_id": "anthropic/claude-3-5-sonnet-20241022",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 78.3,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5 Nano",
   "model_id": "openai/gpt-5-nano",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 78.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Magistral Small",
   "model_id": "mistral/magistral-small",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 78.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Magistral Small 2506",
   "model_id": "mistral/magistral-small-2506",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 78.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 27B",
   "model_id": "google/gemma-3-27b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 77.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2,, domain-evals llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 32B Instruct",
   "model_id": "qwen/qwen2-5-32b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 76.8,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 32B Instruct AWQ",
   "model_id": "qwen/qwen2-5-32b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 76.8,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o",
   "model_id": "openai/gpt-4o",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 76.6,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-05-13)",
   "model_id": "openai/gpt-4o-2024-05-13",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 76.6,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-08-06)",
   "model_id": "openai/gpt-4o-2024-08-06",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 76.6,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-11-20)",
   "model_id": "openai/gpt-4o-2024-11-20",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 76.6,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Scout 17B 16E",
   "model_id": "meta/llama-4-scout-17b-16e",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 76.5,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Scout 17B 16E Instruct",
   "model_id": "meta/llama-4-scout-17b-16e-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 76.5,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-4-Scout-17B-16E-Instruct-FP8",
   "model_id": "meta/llama-4-scout-17b-16e-instruct-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 76.5,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Qwen 7B",
   "model_id": "deepseek/deepseek-r1-distill-qwen-7b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 75.3,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 8B",
   "model_id": "qwen/qwen3-8b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 74.8,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 8B AWQ",
   "model_id": "qwen/qwen3-8b-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 74.8,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 8B Base",
   "model_id": "qwen/qwen3-8b-base",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 74.8,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Pro",
   "model_id": "google/gemini-1-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 74.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 405B",
   "model_id": "meta/llama-3-1-405b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 73.8,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 405B FP8",
   "model_id": "meta/llama-3-1-405b-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 73.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 405B Instruct",
   "model_id": "meta/llama-3-1-405b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 73.8,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 405B Instruct FP8",
   "model_id": "meta/llama-3-1-405b-instruct-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 73.8,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.3 70B Instruct NVFP4",
   "model_id": "nvidia/llama-3-3-70b-instruct-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 73.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-3.3-70B-Instruct",
   "model_id": "meta/llama-3-3-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 73.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Command A",
   "model_id": "cohere/command-a-03-2025",
   "provider": "cohere",
   "provider_display": "Cohere",
   "score": 72.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Command A Reasoning",
   "model_id": "cohere/command-a-reasoning-08-2025",
   "provider": "cohere",
   "provider_display": "Cohere",
   "score": 72.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Phi 4 mini instruct",
   "model_id": "microsoft/phi-4-mini-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 72.5,
   "source": "lmarena.ai, provider-reports, preference-evals, llm-stats, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4 Turbo",
   "model_id": "openai/gpt-4-turbo",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 72.2,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Llama 8B",
   "model_id": "deepseek/deepseek-r1-distill-llama-8b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 72.1,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.0 Flash Lite",
   "model_id": "google/gemini-2-0-flash-lite",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 72.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2",
   "model_id": "xai/grok-2",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 72,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 (1212)",
   "model_id": "xai/grok-2-1212",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 72,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Latest",
   "model_id": "xai/grok-2-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 72,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Vision",
   "model_id": "xai/grok-2-vision",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 72,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Vision (1212)",
   "model_id": "xai/grok-2-vision-1212",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 72,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Vision Latest",
   "model_id": "xai/grok-2-vision-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 72,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1 nano",
   "model_id": "openai/gpt-4-1-nano",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 70.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 32B Instruct",
   "model_id": "qwen/qwen2-5-coder-32b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 70.5,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 32B Instruct AWQ",
   "model_id": "qwen/qwen2-5-coder-32b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 70.5,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o mini",
   "model_id": "openai/gpt-4o-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 70.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3.5",
   "model_id": "anthropic/claude-3-5-haiku-20241022",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 69.5,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3.5 (latest)",
   "model_id": "anthropic/claude-3-5-haiku-latest",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 69.5,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2 27B it",
   "model_id": "google/gemma-2-27b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 68.5,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 14B Instruct",
   "model_id": "qwen/qwen2-5-14b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 68.5,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 14B Instruct AWQ",
   "model_id": "qwen/qwen2-5-14b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 68.5,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Flash",
   "model_id": "google/gemini-1-5-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 65.8,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Flash-8B",
   "model_id": "google/gemini-1-5-flash-8b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 65.8,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 12B",
   "model_id": "google/gemma-3-12b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 65.3,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 4B",
   "model_id": "qwen/qwen3-4b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 65.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 4B Base",
   "model_id": "qwen/qwen3-4b-base",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 65.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 4B Instruct 2507",
   "model_id": "qwen/qwen3-4b-instruct-2507",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 65.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 4B Instruct 2507 FP8",
   "model_id": "qwen/qwen3-4b-instruct-2507-fp8",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 65.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Codestral (latest)",
   "model_id": "mistral/codestral-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 65.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Command R+",
   "model_id": "cohere/command-r-plus-08-2024",
   "provider": "cohere",
   "provider_display": "Cohere",
   "score": 65.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 70B",
   "model_id": "meta/llama-3-1-70b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 64.5,
   "source": "lmarena.ai, provider-reports, safety-evals, domain-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 70B Instruct",
   "model_id": "meta/llama-3-1-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 64.5,
   "source": "lmarena.ai, provider-reports, safety-evals, domain-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 3",
   "model_id": "anthropic/claude-3-opus-20240229",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 60.1,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 7B",
   "model_id": "qwen/qwen2-5-7b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 58.5,
   "source": "provider-reports, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 7B Instruct",
   "model_id": "qwen/qwen2-5-7b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 58.5,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 7B Instruct AWQ",
   "model_id": "qwen/qwen2-5-7b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 58.5,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B",
   "model_id": "01-ai/yi-1-5-34b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 58.5,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B Chat",
   "model_id": "01-ai/yi-1-5-34b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 58.5,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x22B",
   "model_id": "mistral/open-mixtral-8x22b",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 56.2,
   "source": "open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x22B Instruct v0.1",
   "model_id": "mistral/mixtral-8x22b-instruct-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 56.2,
   "source": "open-llm-leaderboard-v2, llm-stats, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Command R",
   "model_id": "cohere/command-r-08-2024",
   "provider": "cohere",
   "provider_display": "Cohere",
   "score": 55.8,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3",
   "model_id": "anthropic/claude-3-sonnet-20240229",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 55.7,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Nemo",
   "model_id": "mistral/mistral-nemo",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 55.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Nemo Base 2407",
   "model_id": "mistral/mistral-nemo-base-2407",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 55.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Nemo Instruct 2407",
   "model_id": "mistral/mistral-nemo-instruct-2407",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 55.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3.5 mini instruct",
   "model_id": "microsoft/phi-3-5-mini-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 52.8,
   "source": "open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2 9B",
   "model_id": "google/gemma-2-9b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 52.1,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2 9B it",
   "model_id": "google/gemma-2-9b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 52.1,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3",
   "model_id": "anthropic/claude-3-haiku-20240307",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 50.3,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 4B",
   "model_id": "google/gemma-3-4b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 48.5,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 3 4B pt",
   "model_id": "google/gemma-3-4b-pt",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 48.5,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "granite 3.1 8B instruct",
   "model_id": "ibm/granite-3-1-8b-instruct",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 46.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4",
   "model_id": "openai/gpt-4",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 42.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B",
   "model_id": "meta/llama-3-1-8b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 42,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct",
   "model_id": "meta/llama-3-1-8b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 42,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct",
   "model_id": "unsloth/llama-3-1-8b-instruct",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 42,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct FP8",
   "model_id": "nvidia/llama-3-1-8b-instruct-fp8",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 42,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct NVFP4",
   "model_id": "nvidia/llama-3-1-8b-instruct-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 42,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 10B Base",
   "model_id": "tii/falcon3-10b-base",
   "provider": "tii",
   "provider_display": "TII",
   "score": 41.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 10B Instruct",
   "model_id": "tii/falcon3-10b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 41.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 7B Instruct",
   "model_id": "tii/falcon3-7b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 40.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x7B",
   "model_id": "mistral/open-mixtral-8x7b",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 40.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x7B Instruct v0.1",
   "model_id": "mistral/mixtral-8x7b-instruct-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 40.1,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x7B v0.1",
   "model_id": "mistral/mixtral-8x7b-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 40.1,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 3B Instruct",
   "model_id": "qwen/qwen2-5-3b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 36.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-3.5-turbo",
   "model_id": "openai/gpt-3-5-turbo",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 32.8,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 14B Instruct",
   "model_id": "qwen/qwen2-5-coder-14b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 32.5,
   "source": "bigcode-leaderboard, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 Mamba 7B Instruct",
   "model_id": "tii/falcon3-mamba-7b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 30.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Math 1.5B",
   "model_id": "qwen/qwen2-5-math-1-5b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 26.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 3B Instruct",
   "model_id": "tii/falcon3-3b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 25.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B Instruct",
   "model_id": "meta/meta-llama-3-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 24.5,
   "source": "llm-stats, open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B Instruct",
   "model_id": "nous-research/meta-llama-3-70b-instruct",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 24.5,
   "source": "llm-stats, open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B Chat",
   "model_id": "01-ai/yi-1-5-9b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 22.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 1.5B Instruct",
   "model_id": "qwen/qwen2-5-1-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 22.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B Chat 16K",
   "model_id": "01-ai/yi-1-5-34b-chat-16k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 21.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 VL 7B Instruct",
   "model_id": "qwen/qwen2-vl-7b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 19.9,
   "source": "open-llm-leaderboard-v2, multimodal-evals"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 VL 7B Instruct AWQ",
   "model_id": "qwen/qwen2-vl-7b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 19.9,
   "source": "open-llm-leaderboard-v2, multimodal-evals"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Ministral 8B Instruct 2410",
   "model_id": "mistral/ministral-8b-instruct-2410",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 19.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 7B Base",
   "model_id": "tii/falcon3-7b-base",
   "provider": "tii",
   "provider_display": "TII",
   "score": 19.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 7B Instruct GPTQ Int4",
   "model_id": "qwen/qwen2-5-coder-7b-instruct-gptq-int4",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 19.2,
   "source": "bigcode-leaderboard, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B",
   "model_id": "meta/meta-llama-3-70b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 18.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B Chat 16K",
   "model_id": "01-ai/yi-1-5-9b-chat-16k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 17.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 3B Instruct",
   "model_id": "meta/llama-3-2-3b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 17.7,
   "source": "llm-stats, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Qwen 1.5B",
   "model_id": "deepseek/deepseek-r1-distill-qwen-1-5b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 16.9,
   "source": "bigcode-leaderboard, provider-reports, open-llm-leaderboard, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Infinity Instruct 3M 0625 Yi 1.5 9B",
   "model_id": "baai/infinity-instruct-3m-0625-yi-1-5-9b",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 16.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3 mini 4K instruct",
   "model_id": "microsoft/phi-3-mini-4k-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 16.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 6B Chat",
   "model_id": "01-ai/yi-1-5-6b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 16.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B 32K",
   "model_id": "01-ai/yi-1-5-34b-32k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 15.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "granite 3.1 2B instruct",
   "model_id": "ibm/granite-3-1-2b-instruct",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 15.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OLMo 2 1124 7B",
   "model_id": "allen-ai/olmo-2-1124-7b",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 14.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "granite 3.0 8B instruct",
   "model_id": "ibm/granite-3-0-8b-instruct",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 14.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3 mini 128K instruct",
   "model_id": "microsoft/phi-3-mini-128k-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 14.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Infinity Instruct 7M Gen Llama3 1 8B",
   "model_id": "baai/infinity-instruct-7m-gen-llama3-1-8b",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 12.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes 2 Mixtral 8x7B DPO",
   "model_id": "nous-research/nous-hermes-2-mixtral-8x7b-dpo",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 12.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 3B Base",
   "model_id": "tii/falcon3-3b-base",
   "provider": "tii",
   "provider_display": "TII",
   "score": 11.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B",
   "model_id": "01-ai/yi-1-5-9b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 11.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B 32K",
   "model_id": "01-ai/yi-1-5-9b-32k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 10.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 0.5B Instruct",
   "model_id": "qwen/qwen2-5-0-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 10.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 2 Theta Llama 3 8B",
   "model_id": "nous-research/hermes-2-theta-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 9.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B Instruct",
   "model_id": "meta/meta-llama-3-8b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 9.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B Instruct",
   "model_id": "nous-research/meta-llama-3-8b-instruct",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 9.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 1.5B",
   "model_id": "qwen/qwen2-5-1-5b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 9.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 1.5B Instruct AWQ",
   "model_id": "qwen/qwen2-5-1-5b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 9.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OLMoE 1B 7B 0125",
   "model_id": "allen-ai/olmoe-1b-7b-0125",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 9.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OLMoE 1B 7B 0125 Instruct",
   "model_id": "allen-ai/olmoe-1b-7b-0125-instruct",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 9.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Infinity Instruct 3M 0625 Llama3 8B",
   "model_id": "baai/infinity-instruct-3m-0625-llama3-8b",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 8.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "glm 4 9B chat hf",
   "model_id": "zhipu/glm-4-9b-chat-hf",
   "provider": "zhipu",
   "provider_display": "Zhipu AI",
   "score": 8.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 2 Pro Llama 3 8B",
   "model_id": "nous-research/hermes-2-pro-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 8.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 1B Instruct",
   "model_id": "meta/llama-3-2-1b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 8.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 1.5B Instruct",
   "model_id": "qwen/qwen2-1-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 7.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 6B",
   "model_id": "01-ai/yi-1-5-6b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 6.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Yi 9B 200K",
   "model_id": "01-ai/yi-9b-200k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 6.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 1B Instruct",
   "model_id": "tii/falcon3-1b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 6.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 34B Chat",
   "model_id": "01-ai/yi-34b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 6.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B",
   "model_id": "cerebras/llama3-1-8b",
   "provider": "cerebras",
   "provider_display": "Cerebras",
   "score": 5.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes 2 SOLAR 10.7B",
   "model_id": "nous-research/nous-hermes-2-solar-10-7b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 5.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 34B 200K",
   "model_id": "01-ai/yi-34b-200k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 5.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 9B",
   "model_id": "01-ai/yi-9b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 5.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OpenHermes 2.5 Mistral 7B",
   "model_id": "teknium/openhermes-2-5-mistral-7b",
   "provider": "teknium",
   "provider_display": "Teknium",
   "score": 5.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 34B",
   "model_id": "01-ai/yi-34b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 5.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 3 Llama 3.1 8B",
   "model_id": "nous-research/hermes-3-llama-3-1-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 4.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 3 Llama 3.1 8B GGUF",
   "model_id": "nous-research/hermes-3-llama-3-1-8b-gguf",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 4.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "falcon mamba 7B",
   "model_id": "tii/falcon-mamba-7b",
   "provider": "tii",
   "provider_display": "TII",
   "score": 4.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "falcon mamba 7B instruct",
   "model_id": "tii/falcon-mamba-7b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 4.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "falcon mamba 7B instruct Q4 K M GGUF",
   "model_id": "tii/falcon-mamba-7b-instruct-q4-k-m-gguf",
   "provider": "tii",
   "provider_display": "TII",
   "score": 4.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "granite 3.1 1B a400m instruct",
   "model_id": "ibm/granite-3-1-1b-a400m-instruct",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 4.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B",
   "model_id": "meta/meta-llama-3-8b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 4.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B",
   "model_id": "nous-research/meta-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 4.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OpenHermes 2 Mistral 7B",
   "model_id": "teknium/openhermes-2-mistral-7b",
   "provider": "teknium",
   "provider_display": "Teknium",
   "score": 4.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "stablelm zephyr 3B",
   "model_id": "stability/stablelm-zephyr-3b",
   "provider": "stability",
   "provider_display": "Stability AI",
   "score": 4.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Yi Coder 9B",
   "model_id": "01-ai/yi-coder-9b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 4.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Yi Coder 9B Chat",
   "model_id": "01-ai/yi-coder-9b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 4.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 3 Llama 3.2 3B",
   "model_id": "nous-research/hermes-3-llama-3-2-3b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 3.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B Instruct v0.3",
   "model_id": "mistral/mistral-7b-instruct-v0-3",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 3.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 0.5B",
   "model_id": "qwen/qwen2-5-0-5b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 3.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2 2B it",
   "model_id": "google/gemma-2-2b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 3.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 1B Base",
   "model_id": "tii/falcon3-1b-base",
   "provider": "tii",
   "provider_display": "TII",
   "score": 3.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 70B hf",
   "model_id": "meta/llama-2-70b-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 3.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 7B Instruct",
   "model_id": "qwen/qwen2-5-coder-7b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 3.1,
   "source": "bigcode-leaderboard, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2 2B",
   "model_id": "google/gemma-2-2b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 3.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2B",
   "model_id": "google/gemma-2b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 3.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B Instruct v0.2",
   "model_id": "mistral/mistral-7b-instruct-v0-2",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 3.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B v0.3",
   "model_id": "mistral/mistral-7b-v0-3",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 3.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "mistral 7B v0.3 bnb 4bit",
   "model_id": "unsloth/mistral-7b-v0-3-bnb-4bit",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 3.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 7B it",
   "model_id": "google/gemma-7b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 2.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 70B chat hf",
   "model_id": "meta/llama-2-70b-chat-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 2.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B v0.1",
   "model_id": "mistral/mistral-7b-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 2.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "phi 2",
   "model_id": "microsoft/phi-2",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 2.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 0.5B Instruct",
   "model_id": "qwen/qwen2-0-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 2.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 11B",
   "model_id": "tii/falcon-11b",
   "provider": "tii",
   "provider_display": "TII",
   "score": 2.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "granite 3.0 1B a400m base",
   "model_id": "ibm/granite-3-0-1b-a400m-base",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 2.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "deepseek moe 16B base",
   "model_id": "deepseek/deepseek-moe-16b-base",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 2.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "GPT NeoXT Chat Base 20B",
   "model_id": "together/gpt-neoxt-chat-base-20b",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 2.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B Instruct v0.1",
   "model_id": "mistral/mistral-7b-instruct-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 2.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "RedPajama INCITE 7B Instruct",
   "model_id": "together/redpajama-incite-7b-instruct",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 2.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek llm 7B base",
   "model_id": "deepseek/deepseek-llm-7b-base",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 2.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek llm 7B chat",
   "model_id": "deepseek/deepseek-llm-7b-chat",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 2.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 40B instruct",
   "model_id": "tii/falcon-40b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 2.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2B it",
   "model_id": "google/gemma-2b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 2.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B chat hf",
   "model_id": "meta/llama-2-7b-chat-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 2.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B chat hf",
   "model_id": "nous-research/llama-2-7b-chat-hf",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 2.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 3B",
   "model_id": "meta/llama-3-2-3b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 1.9,
   "source": "llm-stats, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 40B",
   "model_id": "tii/falcon-40b",
   "provider": "tii",
   "provider_display": "TII",
   "score": 1.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "gemma 1.1 2B it",
   "model_id": "google/gemma-1-1-2b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 1.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B 200K",
   "model_id": "01-ai/yi-6b-200k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 1.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B hf",
   "model_id": "meta/llama-2-7b-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 1.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B hf",
   "model_id": "nous-research/llama-2-7b-hf",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 1.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "OLMo 1B hf",
   "model_id": "allen-ai/olmo-1b-hf",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 1.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OLMoE 1B 7B 0924",
   "model_id": "allen-ai/olmoe-1b-7b-0924",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 1.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Jamba v0.1",
   "model_id": "ai21/jamba-v0-1",
   "provider": "ai21",
   "provider_display": "AI21 Labs",
   "score": 1.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B 32K Instruct",
   "model_id": "together/llama-2-7b-32k-instruct",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 1.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "RedPajama INCITE 7B Base",
   "model_id": "together/redpajama-incite-7b-base",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 1.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B",
   "model_id": "01-ai/yi-6b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 1.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 13B hf",
   "model_id": "meta/llama-2-13b-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 1.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 13B hf",
   "model_id": "nous-research/llama-2-13b-hf",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 1.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "flan t5 large",
   "model_id": "google/flan-t5-large",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 1.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 13B chat hf",
   "model_id": "meta/llama-2-13b-chat-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 1.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "LLaMA 2 7B 32K",
   "model_id": "together/llama-2-7b-32k",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 1.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "RedPajama INCITE Base 3B v1",
   "model_id": "together/redpajama-incite-base-3b-v1",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 1.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B Chat",
   "model_id": "01-ai/yi-6b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 1.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B Chat 4bits",
   "model_id": "01-ai/yi-6b-chat-4bits",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 1.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 7B instruct",
   "model_id": "tii/falcon-7b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 1.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 1B",
   "model_id": "meta/llama-3-2-1b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 1.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 1B",
   "model_id": "nous-research/llama-3-2-1b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 1.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OpenHermes 13B",
   "model_id": "teknium/openhermes-13b",
   "provider": "teknium",
   "provider_display": "Teknium",
   "score": 1.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "flan t5 base",
   "model_id": "google/flan-t5-base",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 1.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "GPT JT 6B v1",
   "model_id": "together/gpt-jt-6b-v1",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 1.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "stablelm 3B 4e1t",
   "model_id": "stability/stablelm-3b-4e1t",
   "provider": "stability",
   "provider_display": "Stability AI",
   "score": 1.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 7B",
   "model_id": "tii/falcon-7b",
   "provider": "tii",
   "provider_display": "TII",
   "score": 1.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "mt5 base",
   "model_id": "google/mt5-base",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 0.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes llama 2 7B",
   "model_id": "nous-research/nous-hermes-llama-2-7b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 0.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "RedPajama INCITE Chat 3B v1",
   "model_id": "together/redpajama-incite-chat-3b-v1",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 0.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "flan t5 small",
   "model_id": "google/flan-t5-small",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 0.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Ministral 3B (latest)",
   "model_id": "mistral/ministral-3b-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 0.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "stablelm 2 1 6B",
   "model_id": "stability/stablelm-2-1-6b",
   "provider": "stability",
   "provider_display": "Stability AI",
   "score": 0.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "RedPajama INCITE 7B Chat",
   "model_id": "together/redpajama-incite-7b-chat",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 0.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "rwkv raven 14B",
   "model_id": "rwkv/rwkv-raven-14b",
   "provider": "rwkv",
   "provider_display": "RWKV Foundation",
   "score": 0.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "flan t5 xl",
   "model_id": "google/flan-t5-xl",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 0.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "DialoGPT medium",
   "model_id": "microsoft/dialogpt-medium",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 0.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "glm 4 9B",
   "model_id": "zhipu/glm-4-9b",
   "provider": "zhipu",
   "provider_display": "Zhipu AI",
   "score": 0.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "GLM 4 9B 0414",
   "model_id": "zhipu/glm-4-9b-0414",
   "provider": "zhipu",
   "provider_display": "Zhipu AI",
   "score": 0.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "glm 4 9B chat",
   "model_id": "zhipu/glm-4-9b-chat",
   "provider": "zhipu",
   "provider_display": "Zhipu AI",
   "score": 0.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "glm 4 9B chat 1M",
   "model_id": "zhipu/glm-4-9b-chat-1m",
   "provider": "zhipu",
   "provider_display": "Zhipu AI",
   "score": 0.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "mt5 small",
   "model_id": "google/mt5-small",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 0.0,
   "source": "open-llm-leaderboard-v2"
  }
 ],
 "page": {
  "aliases": [
   "MATH500",
   "hendrycks_math500",
   "minerva_math500"
  ],
  "category": "math",
  "contamination": {
   "note": "Answers and solutions are distributed in the open dataset. The source MATH corpus has circulated since 2021 and was the subject of a January 2025 DMCA takedown by Art of Problem Solving alleging over 10,000 of its 12,500 problems were copied from AoPS's Alcumus platform. Wu et al. 2025 (arXiv 2507.10532) report memorization-consistent behaviour by Qwen2.5-series models specifically on MATH-500.",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 500,
   "size_note": "500 problems held out from the original MATH test split of 5,000; the other 4,500 MATH test problems were moved into the training set for this evaluation.",
   "splits": "single 'test' split, 500 rows",
   "url": "https://huggingface.co/datasets/HuggingFaceH4/MATH-500"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice G",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "hendrycks_math500, minerva_math500",
   "opencompass": "",
   "other": "OpenAI's openai/simple-evals loads MATH-500 via the math_500_test split in math_eval.py."
  },
  "id": "math_500",
  "last_updated": "",
  "leaderboard_url": "https://artificialanalysis.ai/evaluations/math-500",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "MATH-500 gives a model 500 written competition mathematics problems spanning algebra, geometry, number theory, counting and probability, precalculus, intermediate algebra and prealgebra. The model reads a problem statement in English and must produce a worked solution ending in a final answer; there are no answer choices. It tests multi-step symbolic and numeric reasoning and correct execution of a solution plan, not speed, tool use, or any skill beyond English-language mathematics.\n",
  "metric": {
   "baseline_note": "No confirmed random-guess or human baseline for this specific 500-item split; free-response grading has no meaningful chance rate.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy (pass@1)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "MATH-500",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2305.20050",
   "title": "Let's Verify Step by Step",
   "url": "https://arxiv.org/abs/2305.20050",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Hunter Lightman",
    "Vineet Kosaraju",
    "Yura Burda",
    "Harri Edwards",
    "Bowen Baker",
    "Teddy Lee",
    "Jan Leike",
    "John Schulman",
    "Ilya Sutskever",
    "Karl Cobbe"
   ],
   "org": "OpenAI",
   "url": "https://github.com/openai/prm800k"
  },
  "released": "2023-05",
  "repo_url": "https://github.com/openai/prm800k",
  "saturation": {
   "as_of": "2026-09",
   "note": "Artificial Analysis's independently run leaderboard has GPT-5 (high) at 99.4%, with o3, Grok 3 mini Reasoning (high), GPT-5 (medium) and Claude 4 Sonnet all at 99.1-99.2%; top models are separated by tenths of a point.",
   "status": "saturated",
   "top_score": 99.4
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "Let's Verify Step by Step (Lightman et al., 2023)",
    "url": "https://arxiv.org/abs/2305.20050"
   },
   {
    "accessed": "2026-09-07",
    "title": "Measuring Mathematical Problem Solving With the MATH Dataset (Hendrycks et al., 2021)",
    "url": "https://arxiv.org/abs/2103.03874"
   },
   {
    "accessed": "2026-09-07",
    "title": "openai/prm800k GitHub repository (math_splits/test.jsonl, MIT licence)",
    "url": "https://github.com/openai/prm800k"
   },
   {
    "accessed": "2026-09-07",
    "title": "HuggingFaceH4/MATH-500 dataset card",
    "url": "https://huggingface.co/datasets/HuggingFaceH4/MATH-500"
   },
   {
    "accessed": "2026-09-07",
    "title": "hendrycks/math GitHub repository",
    "url": "https://github.com/hendrycks/math"
   },
   {
    "accessed": "2026-09-07",
    "title": "hendrycks/competition_math dataset page (shows DMCA takedown notice)",
    "url": "https://huggingface.co/datasets/hendrycks/competition_math"
   },
   {
    "accessed": "2026-09-07",
    "title": "Hugging Face legal takedown notice, Art of Problem Solving vs. hendrycks/competition_math",
    "url": "https://huggingface.co/datasets/huggingface-legal/takedown-notices/blob/main/2025/2025-01-02-AoPS.md"
   },
   {
    "accessed": "2026-09-07",
    "title": "openai/simple-evals math_eval.py",
    "url": "https://github.com/openai/simple-evals/blob/main/math_eval.py"
   },
   {
    "accessed": "2026-09-07",
    "title": "openai/simple-evals README",
    "url": "https://github.com/openai/simple-evals/blob/main/README.md"
   },
   {
    "accessed": "2026-09-07",
    "title": "MATH-500 Benchmark Leaderboard | Artificial Analysis",
    "url": "https://artificialanalysis.ai/evaluations/math-500"
   },
   {
    "accessed": "2026-09-07",
    "title": "Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination (Wu et al., 2025)",
    "url": "https://arxiv.org/abs/2507.10532"
   }
  ],
  "status": "active",
  "subcategory": "competition mathematics",
  "summary": "A fixed 500-problem subset of the MATH test set used to grade free-response competition mathematics after the rest of the split was moved into training data.",
  "tags": [
   "math",
   "competition-math",
   "chain-of-thought",
   "free-response"
  ],
  "task_format": "Free-response: read a competition math problem, output a worked solution and a final answer (typically after 'Answer:' or boxed)."
 }
}