{
 "body": "\n## What it measures\n\nGPQA Diamond tests whether a model can answer graduate-level multiple-choice questions in biology,\nphysics and chemistry that are deliberately built to resist internet lookup. Each question was written\nand validated by a PhD-level domain expert, then filtered so that expert annotators agreed on the answer\nwhile most skilled non-experts, given 30-plus minutes and open web access, did not. It is a single-turn,\ntext-only, English-language task: four answer options, one correct letter.\n\nThe \"Google-proof\" framing is the point. Where many knowledge benchmarks reward retrieval of a fact a\nsearch engine could surface directly, GPQA Diamond is filtered specifically against that failure mode,\nso a high score is meant to reflect domain reasoning rather than lookup skill \u2014 which is also why this\npage treats it as closer to reasoning than to plain recall-style knowledge testing.\n\n## How it is scored\n\nModels are graded on accuracy: the percentage of the 198 questions answered with the correct letter.\nRandom guessing scores 25% on the four-option format. Most labs now report a zero-shot or few-shot\nchain-of-thought variant, prompting the model to reason before committing to an answer; OpenAI's widely\nreused simple-evals prompt is the de facto standard, with answers parsed from a final \"ANSWER: X\" line.\nBecause parsing is strict, a model that reasons correctly but formats its answer wrong can score below\nthe random baseline, which Epoch AI has documented happening in its own runs.\n\n## Dataset and licence\n\nGPQA Diamond is the 198-question hardest tier of the 448-question main GPQA set, produced by recruiting\nsubject-matter experts (people with or pursuing PhDs) to write questions, then having both other experts\nand skilled non-expert validators attempt each one; Diamond keeps only the questions where both experts\nsucceeded and most non-experts failed. The dataset is released under CC BY 4.0 on Hugging Face.\nDistribution is deliberately gated: the GitHub copy is password-protected, and downloaders agree not to\npublicly post questions, specifically to slow the benchmark's entry into future training data. English\nonly, text only.\n\n## Who publishes it\n\nGPQA was introduced by David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe\nPang, Julien Dirani, Julian Michael and Samuel R. Bowman, posted to arXiv in November 2023. The authors\nmaintain the reference dataset and code at github.com/idavidrein/gpqa. Independent trackers, notably\nEpoch AI, run and publish their own GPQA Diamond evaluations of new frontier models as a standing\nbenchmark on their site.\n\n## Lineage\n\nGPQA names no formal predecessor and has no official successor within its own line, but its rapid\nsaturation is one of the stated reasons newer, harder science and reasoning benchmarks exist: OpenAI's\nFrontierScience and Scale AI/CAIS's Humanity's Last Exam were both framed partly as responses to GPQA and\nMMLU losing their power to separate frontier models. Neither has a page in this repository yet. GPQA\nDiamond is one of three official subsets of the original GPQA release, alongside the full \"main\" set and\na broader \"extended\" set; only Diamond is treated as a standalone page here.\n\n## Saturation and contamination\n\nGPQA Diamond has moved from a genuinely hard benchmark to a largely saturated one within two years. GPT-4\nscored 39% at the November 2023 release, against OpenAI's own later-measured PhD-expert baseline of\n69.7%. OpenAI's o1 reached 77.3% in September 2024, already above that expert baseline. Epoch AI measured\nGrok 4 at 87% in July 2025, and OpenAI reported GPT-5.2 at 92% in its December 2025 FrontierScience\nannouncement. That trajectory leaves little headroom before the 100% ceiling. Contamination risk sits at\nmedium: the dataset uses a canary string and gated, agreement-bound distribution specifically to resist\nleakage, but the questions have circulated for almost three years among a research community that both\ntrains on and evaluates against them.\n\n## How to run it\n\nThe authors' own repository ships `run_baseline.py` with zero-shot and chain-of-thought prompting\noptions. EleutherAI's lm-evaluation-harness defines a family of task names per subset and shot count,\nincluding `gpqa_diamond_zeroshot` and `gpqa_diamond_cot_zeroshot`, plus few-shot and generative variants.\nOpenAI's simple-evals package (`gpqa_eval.py`) supplies the specific zero-shot chain-of-thought prompt\nmost frontier-model announcements now use, which makes it the closest thing to a comparability standard\neven though it is not lm-evaluation-harness's own default. Because chain-of-thought budget, answer-parsing\nstrictness and prompt wording all vary between these implementations, scores from different sources are\nnot perfectly comparable.\n\n## Reading the numbers\n\nA high GPQA Diamond score today mostly confirms a model has strong graduate-level scientific recall and\nmulti-step reasoning within a closed four-option format, not that it can do open-ended scientific work.\nBecause the benchmark now sits close to its ceiling, small differences between frontier models are within\nnoise, and it no longer separates the best systems the way it did in 2023 and 2024. Read a GPQA Diamond\nnumber alongside a still-unsaturated benchmark, such as Humanity's Last Exam or a research-style track\nlike FrontierScience-Research, to judge whether a model's scientific reasoning still has headroom, and\ntreat scores from different harnesses or prompt formats as roughly comparable at best.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "gpqa_diamond",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Mythos Preview",
   "model_id": "anthropic/claude-mythos-preview",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 94.55,
   "source": "anthropic-system-card"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 3.1 Pro Preview",
   "model_id": "google/gemini-3-1-pro-preview",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 94.3,
   "source": "anthropic-system-card-mythos, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.4",
   "model_id": "openai/gpt-5-4",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 92.8,
   "source": "lmarena.ai, provider-reports, anthropic-system-card-mythos, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.6",
   "model_id": "anthropic/claude-opus-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 91.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, anthropic-system-card-mythos"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o3-pro",
   "model_id": "openai/o3-pro",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 89,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o3",
   "model_id": "openai/o3",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 87.7,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o3-deep-research",
   "model_id": "openai/o3-deep-research",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 87.7,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GLM 5.1",
   "model_id": "zhipu/glm-5-1",
   "provider": "zhipu",
   "provider_display": "Z.ai (Zhipu AI)",
   "score": 86.2,
   "source": "zai-org-model-card"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o4-mini",
   "model_id": "openai/o4-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 82.3,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o4-mini-deep-research",
   "model_id": "openai/o4-mini-deep-research",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 82.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro",
   "model_id": "google/gemini-2-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 80.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 05-06",
   "model_id": "google/gemini-2-5-pro-preview-05-06",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 80.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 06-05",
   "model_id": "google/gemini-2-5-pro-preview-06-05",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 80.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview TTS",
   "model_id": "google/gemini-2-5-pro-preview-tts",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 80.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o1-pro",
   "model_id": "openai/o1-pro",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 80.5,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o3-mini",
   "model_id": "openai/o3-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 79.7,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.1",
   "model_id": "openai/gpt-5-1",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 79.1,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.1 Chat",
   "model_id": "openai/gpt-5-1-chat-latest",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 79.1,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.1 Codex",
   "model_id": "openai/gpt-5-1-codex",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 79.1,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.1 Codex Max",
   "model_id": "openai/gpt-5-1-codex-max",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 79.1,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.1 Codex mini",
   "model_id": "openai/gpt-5-1-codex-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 79.1,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4",
   "model_id": "anthropic/claude-opus-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 78.4,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5",
   "model_id": "openai/gpt-5",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 77.2,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5 Chat (latest)",
   "model_id": "openai/gpt-5-chat-latest",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 77.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5 Pro",
   "model_id": "openai/gpt-5-pro",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 77.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5-Codex",
   "model_id": "openai/gpt-5-codex",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 77.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.2",
   "model_id": "openai/gpt-5-2",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 77.2,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.2 Chat",
   "model_id": "openai/gpt-5-2-chat-latest",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 77.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.2 Codex",
   "model_id": "openai/gpt-5-2-codex",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 77.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.2 Pro",
   "model_id": "openai/gpt-5-2-pro",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 77.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.3 Chat (latest)",
   "model_id": "openai/gpt-5-3-chat-latest",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 77.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.3 Codex",
   "model_id": "openai/gpt-5-3-codex",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 77.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.3 Codex Spark",
   "model_id": "openai/gpt-5-3-codex-spark",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 77.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.4 mini",
   "model_id": "openai/gpt-5-4-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 77.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.4 nano",
   "model_id": "openai/gpt-5-4-nano",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 77.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.4 Pro",
   "model_id": "openai/gpt-5-4-pro",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 77.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.5",
   "model_id": "anthropic/claude-opus-4-5-20251101",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 77.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.5 (latest)",
   "model_id": "anthropic/claude-opus-4-5",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 77.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.1",
   "model_id": "anthropic/claude-opus-4-1-20250805",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 76.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.1 (latest)",
   "model_id": "anthropic/claude-opus-4-1",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 76.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4",
   "model_id": "xai/grok-4",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 75.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4 Fast",
   "model_id": "xai/grok-4-fast",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 75.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4 Fast (Non-Reasoning)",
   "model_id": "xai/grok-4-fast-non-reasoning",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 75.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4.1 Fast",
   "model_id": "xai/grok-4-1-fast",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 75.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4.1 Fast (Non-Reasoning)",
   "model_id": "xai/grok-4-1-fast-non-reasoning",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 75.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4.20 (Non-Reasoning)",
   "model_id": "xai/grok-4-20-0309-non-reasoning",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 75.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4.20 (Reasoning)",
   "model_id": "xai/grok-4-20-0309-reasoning",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 75.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4.20 Multi-Agent",
   "model_id": "xai/grok-4-20-multi-agent-0309",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 75.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4",
   "model_id": "anthropic/claude-sonnet-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 74.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.6",
   "model_id": "anthropic/claude-sonnet-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 74.1,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4 (latest)",
   "model_id": "anthropic/claude-opus-4-0",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 73.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o1",
   "model_id": "openai/o1",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 73.3,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o1-preview",
   "model_id": "openai/o1-preview",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 73.3,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 0528",
   "model_id": "deepseek/deepseek-r1-0528",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 73.2,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 0528 NVFP4 v2",
   "model_id": "nvidia/deepseek-r1-0528-nvfp4-v2",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 73.2,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.5",
   "model_id": "anthropic/claude-sonnet-4-5-20250929",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 72.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.5 (latest)",
   "model_id": "anthropic/claude-sonnet-4-5",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 72.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1",
   "model_id": "deepseek/deepseek-r1",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 71.5,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek Reasoner",
   "model_id": "deepseek/deepseek-reasoner",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 71.5,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 235B-A22B",
   "model_id": "qwen/qwen3-235b-a22b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 71.1,
   "source": "lmarena.ai, provider-reports, preference-evals, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3-Coder 480B-A35B Instruct",
   "model_id": "qwen/qwen3-coder-480b-a35b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 70.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash",
   "model_id": "google/gemini-2-5-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 70.2,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Image",
   "model_id": "google/gemini-2-5-flash-image",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 70.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Image (Preview)",
   "model_id": "google/gemini-2-5-flash-image-preview",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 70.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Lite",
   "model_id": "google/gemini-2-5-flash-lite",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 70.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Lite Preview 06-17",
   "model_id": "google/gemini-2-5-flash-lite-preview-06-17",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 70.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Lite Preview 09-25",
   "model_id": "google/gemini-2-5-flash-lite-preview-09-2025",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 70.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Preview 04-17",
   "model_id": "google/gemini-2-5-flash-preview-04-17",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 70.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Preview 05-20",
   "model_id": "google/gemini-2-5-flash-preview-05-20",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 70.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Preview 09-25",
   "model_id": "google/gemini-2-5-flash-preview-09-2025",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 70.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Preview TTS",
   "model_id": "google/gemini-2-5-flash-preview-tts",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 70.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1",
   "model_id": "openai/gpt-4-1",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 69.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, domain-evals preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4 (latest)",
   "model_id": "anthropic/claude-sonnet-4-0",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 68.4,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5 Mini",
   "model_id": "openai/gpt-5-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 68.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.7",
   "model_id": "anthropic/claude-3-7-sonnet-20250219",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 68.0,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 31B",
   "model_id": "google/gemma-4-31b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 67.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 31B it",
   "model_id": "google/gemma-4-31b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 67.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 31B it GGUF",
   "model_id": "unsloth/gemma-4-31b-it-gguf",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 67.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 31B IT NVFP4",
   "model_id": "nvidia/gemma-4-31b-it-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 67.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 26B",
   "model_id": "google/gemma-4-26b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 65.2,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 26B A4B it",
   "model_id": "google/gemma-4-26b-a4b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 65.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 26B A4B it GGUF",
   "model_id": "unsloth/gemma-4-26b-a4b-it-gguf",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 65.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5",
   "model_id": "anthropic/claude-3-5-sonnet-20240620",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 65.0,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5 v2",
   "model_id": "anthropic/claude-3-5-sonnet-20241022",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 65.0,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3.2",
   "model_id": "deepseek/deepseek-v3-2",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 64.5,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3.2 Exp",
   "model_id": "deepseek/deepseek-v3-2-exp",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 64.5,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3",
   "model_id": "xai/grok-3",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 62.8,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3 Fast",
   "model_id": "xai/grok-3-fast",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 62.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3 Fast Latest",
   "model_id": "xai/grok-3-fast-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 62.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3 Latest",
   "model_id": "xai/grok-3-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 62.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Maverick 17B 128E Instruct",
   "model_id": "meta/llama-4-maverick-17b-128e-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 62.5,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-4-Maverick-17B-128E-Instruct-FP8",
   "model_id": "meta/llama-4-maverick-17b-128e-instruct-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 62.5,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 32B",
   "model_id": "qwen/qwen3-32b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 62.5,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 32B AWQ",
   "model_id": "qwen/qwen3-32b-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 62.5,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 32B NVFP4",
   "model_id": "nvidia/qwen3-32b-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 62.5,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3.1",
   "model_id": "deepseek/deepseek-v3-1",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 62.3,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 4.5",
   "model_id": "anthropic/claude-haiku-4-5-20251001",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 62.1,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 4.5 (latest)",
   "model_id": "anthropic/claude-haiku-4-5",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 62.1,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Magistral Medium (latest)",
   "model_id": "mistral/magistral-medium-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 60.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.0 Flash",
   "model_id": "google/gemini-2-0-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 60.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1 mini",
   "model_id": "openai/gpt-4-1-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 60.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o1-mini",
   "model_id": "openai/o1-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 60.0,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 3",
   "model_id": "anthropic/claude-3-opus-20240229",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 59.4,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek Chat",
   "model_id": "deepseek/deepseek-chat",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 59.1,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V2",
   "model_id": "deepseek/deepseek-v2",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 59.1,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V2 Lite",
   "model_id": "deepseek/deepseek-v2-lite",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 59.1,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V2 Lite Chat",
   "model_id": "deepseek/deepseek-v2-lite-chat",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 59.1,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3",
   "model_id": "deepseek/deepseek-v3",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 59.1,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, domain-evals open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3 0324",
   "model_id": "deepseek/deepseek-v3-0324",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 59.1,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "QwQ Plus",
   "model_id": "qwen/qwq-plus",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 58.5,
   "source": "provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 27B",
   "model_id": "google/gemma-3-27b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 58.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2,, domain-evals llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Pro",
   "model_id": "google/gemini-1-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 55.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Scout 17B 16E",
   "model_id": "meta/llama-4-scout-17b-16e",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 55.8,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Scout 17B 16E Instruct",
   "model_id": "meta/llama-4-scout-17b-16e-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 55.8,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-4-Scout-17B-16E-Instruct-FP8",
   "model_id": "meta/llama-4-scout-17b-16e-instruct-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 55.8,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large (latest)",
   "model_id": "mistral/mistral-large-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 55.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large 2.1",
   "model_id": "mistral/mistral-large-2411",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 55.5,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals,, domain-evals open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large 3",
   "model_id": "mistral/mistral-large-2512",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 55.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3 Mini",
   "model_id": "xai/grok-3-mini",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 55.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3 Mini Fast",
   "model_id": "xai/grok-3-mini-fast",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 55.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3 Mini Fast Latest",
   "model_id": "xai/grok-3-mini-fast-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 55.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3 Mini Latest",
   "model_id": "xai/grok-3-mini-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 55.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Llama 70B",
   "model_id": "deepseek/deepseek-r1-distill-llama-70b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 55.2,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 30B A3B Instruct 2507",
   "model_id": "qwen/qwen3-30b-a3b-instruct-2507",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 55.2,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 30B A3B NVFP4",
   "model_id": "nvidia/qwen3-30b-a3b-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 55.2,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 30B-A3B",
   "model_id": "qwen/qwen3-30b-a3b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 55.2,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5 Nano",
   "model_id": "openai/gpt-5-nano",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 55.1,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 72B Instruct",
   "model_id": "qwen/qwen2-5-72b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 54.8,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o",
   "model_id": "openai/gpt-4o",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 53.6,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-05-13)",
   "model_id": "openai/gpt-4o-2024-05-13",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 53.6,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-08-06)",
   "model_id": "openai/gpt-4o-2024-08-06",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 53.6,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-11-20)",
   "model_id": "openai/gpt-4o-2024-11-20",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 53.6,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Qwen 32B",
   "model_id": "deepseek/deepseek-r1-distill-qwen-32b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 52.8,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "phi 4",
   "model_id": "microsoft/phi-4",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 52.5,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2,, domain-evals llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Phi 4 multimodal instruct",
   "model_id": "microsoft/phi-4-multimodal-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 52.5,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3.5",
   "model_id": "anthropic/claude-3-5-haiku-20241022",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 51.2,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3.5 (latest)",
   "model_id": "anthropic/claude-3-5-haiku-latest",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 51.2,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Magistral Small",
   "model_id": "mistral/magistral-small",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 50.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Magistral Small 2506",
   "model_id": "mistral/magistral-small-2506",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 50.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.3 70B Instruct NVFP4",
   "model_id": "nvidia/llama-3-3-70b-instruct-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 50.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-3.3-70B-Instruct",
   "model_id": "meta/llama-3-3-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 50.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2",
   "model_id": "xai/grok-2",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 50.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 (1212)",
   "model_id": "xai/grok-2-1212",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 50.1,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Latest",
   "model_id": "xai/grok-2-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 50.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Vision",
   "model_id": "xai/grok-2-vision",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 50.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Vision (1212)",
   "model_id": "xai/grok-2-vision-1212",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 50.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Vision Latest",
   "model_id": "xai/grok-2-vision-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 50.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 14B",
   "model_id": "qwen/qwen3-14b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 50.1,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 14B AWQ",
   "model_id": "qwen/qwen3-14b-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 50.1,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 14B NVFP4",
   "model_id": "nvidia/qwen3-14b-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 50.1,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 405B",
   "model_id": "meta/llama-3-1-405b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 49.8,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 405B FP8",
   "model_id": "meta/llama-3-1-405b-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 49.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 405B Instruct",
   "model_id": "meta/llama-3-1-405b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 49.8,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 405B Instruct FP8",
   "model_id": "meta/llama-3-1-405b-instruct-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 49.8,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4 Turbo",
   "model_id": "openai/gpt-4-turbo",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 49.3,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.0 Flash Lite",
   "model_id": "google/gemini-2-0-flash-lite",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 48.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 32B Instruct",
   "model_id": "qwen/qwen2-5-32b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 48.5,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 32B Instruct AWQ",
   "model_id": "qwen/qwen2-5-32b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 48.5,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1 nano",
   "model_id": "openai/gpt-4-1-nano",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 48.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Command A",
   "model_id": "cohere/command-a-03-2025",
   "provider": "cohere",
   "provider_display": "Cohere",
   "score": 48.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Command A Reasoning",
   "model_id": "cohere/command-a-reasoning-08-2025",
   "provider": "cohere",
   "provider_display": "Cohere",
   "score": 48.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2 27B it",
   "model_id": "google/gemma-2-27b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 48.1,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o mini",
   "model_id": "openai/gpt-4o-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 46.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3",
   "model_id": "anthropic/claude-3-sonnet-20240229",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 45.5,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 12B",
   "model_id": "google/gemma-3-12b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 45.5,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Qwen 14B",
   "model_id": "deepseek/deepseek-r1-distill-qwen-14b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 45.3,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Flash",
   "model_id": "google/gemini-1-5-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 45.2,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Flash-8B",
   "model_id": "google/gemini-1-5-flash-8b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 45.2,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Codestral (latest)",
   "model_id": "mistral/codestral-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 42.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Command R+",
   "model_id": "cohere/command-r-plus-08-2024",
   "provider": "cohere",
   "provider_display": "Cohere",
   "score": 42.5,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 70B",
   "model_id": "meta/llama-3-1-70b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 42.5,
   "source": "lmarena.ai, provider-reports, safety-evals, domain-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 70B Instruct",
   "model_id": "meta/llama-3-1-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 42.5,
   "source": "lmarena.ai, provider-reports, safety-evals, domain-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 32B Instruct",
   "model_id": "qwen/qwen2-5-coder-32b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 42.5,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 32B Instruct AWQ",
   "model_id": "qwen/qwen2-5-coder-32b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 42.5,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 8B",
   "model_id": "qwen/qwen3-8b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 42.5,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 8B AWQ",
   "model_id": "qwen/qwen3-8b-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 42.5,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 8B Base",
   "model_id": "qwen/qwen3-8b-base",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 42.5,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Phi 4 mini instruct",
   "model_id": "microsoft/phi-4-mini-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 42.1,
   "source": "lmarena.ai, provider-reports, preference-evals, llm-stats, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 14B Instruct",
   "model_id": "qwen/qwen2-5-14b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 42.1,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 14B Instruct AWQ",
   "model_id": "qwen/qwen2-5-14b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 42.1,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3",
   "model_id": "anthropic/claude-3-haiku-20240307",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 41.2,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B",
   "model_id": "meta/meta-llama-3-70b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 39.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4",
   "model_id": "openai/gpt-4",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 39.7,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Qwen 7B",
   "model_id": "deepseek/deepseek-r1-distill-qwen-7b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 38.5,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2 9B",
   "model_id": "google/gemma-2-9b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 38.2,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2 9B it",
   "model_id": "google/gemma-2-9b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 38.2,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B",
   "model_id": "01-ai/yi-1-5-9b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 37.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 34B",
   "model_id": "01-ai/yi-34b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 36.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B",
   "model_id": "01-ai/yi-1-5-34b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 36.5,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B Chat",
   "model_id": "01-ai/yi-1-5-34b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 36.5,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B 32K",
   "model_id": "01-ai/yi-1-5-34b-32k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 36.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Llama 8B",
   "model_id": "deepseek/deepseek-r1-distill-llama-8b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 36.2,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x22B",
   "model_id": "mistral/open-mixtral-8x22b",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 36.1,
   "source": "open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x22B Instruct v0.1",
   "model_id": "mistral/mixtral-8x22b-instruct-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 36.1,
   "source": "open-llm-leaderboard-v2, llm-stats, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B 32K",
   "model_id": "01-ai/yi-1-5-9b-32k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 35.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 34B 200K",
   "model_id": "01-ai/yi-34b-200k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 35.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Infinity Instruct 3M 0625 Yi 1.5 9B",
   "model_id": "baai/infinity-instruct-3m-0625-yi-1-5-9b",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 35.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Command R",
   "model_id": "cohere/command-r-08-2024",
   "provider": "cohere",
   "provider_display": "Cohere",
   "score": 35.2,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 7B",
   "model_id": "qwen/qwen2-5-7b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 35.2,
   "source": "provider-reports, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 7B Instruct",
   "model_id": "qwen/qwen2-5-7b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 35.2,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 7B Instruct AWQ",
   "model_id": "qwen/qwen2-5-7b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 35.2,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 7B Base",
   "model_id": "tii/falcon3-7b-base",
   "provider": "tii",
   "provider_display": "TII",
   "score": 34.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 4B",
   "model_id": "qwen/qwen3-4b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 34.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 4B Base",
   "model_id": "qwen/qwen3-4b-base",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 34.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 4B Instruct 2507",
   "model_id": "qwen/qwen3-4b-instruct-2507",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 34.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 4B Instruct 2507 FP8",
   "model_id": "qwen/qwen3-4b-instruct-2507-fp8",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 34.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B Chat 16K",
   "model_id": "01-ai/yi-1-5-34b-chat-16k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 33.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 34B Chat",
   "model_id": "01-ai/yi-34b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 33.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Nemo",
   "model_id": "mistral/mistral-nemo",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 33.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Nemo Base 2407",
   "model_id": "mistral/mistral-nemo-base-2407",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 33.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Nemo Instruct 2407",
   "model_id": "mistral/mistral-nemo-instruct-2407",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 33.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B Chat",
   "model_id": "01-ai/yi-1-5-9b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 33.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 4B",
   "model_id": "google/gemma-3-4b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 33.2,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 3 4B pt",
   "model_id": "google/gemma-3-4b-pt",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 33.2,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "granite 3.0 8B instruct",
   "model_id": "ibm/granite-3-0-8b-instruct",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 33.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3 mini 4K instruct",
   "model_id": "microsoft/phi-3-mini-4k-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 32.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes 2 Mixtral 8x7B DPO",
   "model_id": "nous-research/nous-hermes-2-mixtral-8x7b-dpo",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 32.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 VL 7B Instruct",
   "model_id": "qwen/qwen2-vl-7b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 32.0,
   "source": "open-llm-leaderboard-v2, multimodal-evals"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 VL 7B Instruct AWQ",
   "model_id": "qwen/qwen2-vl-7b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 32.0,
   "source": "open-llm-leaderboard-v2, multimodal-evals"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3 mini 128K instruct",
   "model_id": "microsoft/phi-3-mini-128k-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 31.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 9B",
   "model_id": "01-ai/yi-9b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 31.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "glm 4 9B",
   "model_id": "zhipu/glm-4-9b",
   "provider": "zhipu",
   "provider_display": "Zhipu AI",
   "score": 31.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "GLM 4 9B 0414",
   "model_id": "zhipu/glm-4-9b-0414",
   "provider": "zhipu",
   "provider_display": "Zhipu AI",
   "score": 31.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Yi 9B 200K",
   "model_id": "01-ai/yi-9b-200k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 31.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "glm 4 9B chat",
   "model_id": "zhipu/glm-4-9b-chat",
   "provider": "zhipu",
   "provider_display": "Zhipu AI",
   "score": 31.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 6B",
   "model_id": "01-ai/yi-1-5-6b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 31.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "falcon mamba 7B",
   "model_id": "tii/falcon-mamba-7b",
   "provider": "tii",
   "provider_display": "TII",
   "score": 31.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "falcon mamba 7B instruct",
   "model_id": "tii/falcon-mamba-7b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 31.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "falcon mamba 7B instruct Q4 K M GGUF",
   "model_id": "tii/falcon-mamba-7b-instruct-q4-k-m-gguf",
   "provider": "tii",
   "provider_display": "TII",
   "score": 31.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 7B Instruct",
   "model_id": "tii/falcon3-7b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 31.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B",
   "model_id": "cerebras/llama3-1-8b",
   "provider": "cerebras",
   "provider_display": "Cerebras",
   "score": 31.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B Chat 16K",
   "model_id": "01-ai/yi-1-5-9b-chat-16k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 30.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B",
   "model_id": "meta/meta-llama-3-8b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 30.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B",
   "model_id": "nous-research/meta-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 30.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 14B Instruct",
   "model_id": "qwen/qwen2-5-coder-14b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 30.5,
   "source": "bigcode-leaderboard, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 Mamba 7B Instruct",
   "model_id": "tii/falcon3-mamba-7b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 30.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "glm 4 9B chat 1M",
   "model_id": "zhipu/glm-4-9b-chat-1m",
   "provider": "zhipu",
   "provider_display": "Zhipu AI",
   "score": 30.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 2 Theta Llama 3 8B",
   "model_id": "nous-research/hermes-2-theta-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 30.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "glm 4 9B chat hf",
   "model_id": "zhipu/glm-4-9b-chat-hf",
   "provider": "zhipu",
   "provider_display": "Zhipu AI",
   "score": 30.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 70B hf",
   "model_id": "meta/llama-2-70b-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 30.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 6B Chat",
   "model_id": "01-ai/yi-1-5-6b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 30.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "granite 3.1 8B instruct",
   "model_id": "ibm/granite-3-1-8b-instruct",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 29.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 3 Llama 3.1 8B",
   "model_id": "nous-research/hermes-3-llama-3-1-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 29.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 3 Llama 3.1 8B GGUF",
   "model_id": "nous-research/hermes-3-llama-3-1-8b-gguf",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 29.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 3B Base",
   "model_id": "tii/falcon3-3b-base",
   "provider": "tii",
   "provider_display": "TII",
   "score": 29.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2 2B it",
   "model_id": "google/gemma-2-2b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 29.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes 2 SOLAR 10.7B",
   "model_id": "nous-research/nous-hermes-2-solar-10-7b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 29.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 7B Instruct",
   "model_id": "qwen/qwen2-5-coder-7b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 29.4,
   "source": "bigcode-leaderboard, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B Chat",
   "model_id": "01-ai/yi-6b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 29.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B Chat 4bits",
   "model_id": "01-ai/yi-6b-chat-4bits",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 29.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 2 Pro Llama 3 8B",
   "model_id": "nous-research/hermes-2-pro-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 29.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Infinity Instruct 7M Gen Llama3 1 8B",
   "model_id": "baai/infinity-instruct-7m-gen-llama3-1-8b",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 29.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B Instruct",
   "model_id": "meta/meta-llama-3-8b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 29.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B Instruct",
   "model_id": "nous-research/meta-llama-3-8b-instruct",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 29.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B v0.1",
   "model_id": "mistral/mistral-7b-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 29.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B v0.3",
   "model_id": "mistral/mistral-7b-v0-3",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 29.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "mistral 7B v0.3 bnb 4bit",
   "model_id": "unsloth/mistral-7b-v0-3-bnb-4bit",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 29.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 3B Instruct",
   "model_id": "tii/falcon3-3b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 28.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "granite 3.1 2B instruct",
   "model_id": "ibm/granite-3-1-2b-instruct",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 28.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B Instruct",
   "model_id": "meta/meta-llama-3-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 28.7,
   "source": "llm-stats, open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B Instruct",
   "model_id": "nous-research/meta-llama-3-70b-instruct",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 28.7,
   "source": "llm-stats, open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B",
   "model_id": "meta/llama-3-1-8b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 28.5,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct",
   "model_id": "meta/llama-3-1-8b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 28.5,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct",
   "model_id": "unsloth/llama-3-1-8b-instruct",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 28.5,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct FP8",
   "model_id": "nvidia/llama-3-1-8b-instruct-fp8",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 28.5,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct NVFP4",
   "model_id": "nvidia/llama-3-1-8b-instruct-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 28.5,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3.5 mini instruct",
   "model_id": "microsoft/phi-3-5-mini-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 28.5,
   "source": "open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 1.5B",
   "model_id": "qwen/qwen2-5-1-5b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 28.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 1.5B Instruct AWQ",
   "model_id": "qwen/qwen2-5-1-5b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 28.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 7B it",
   "model_id": "google/gemma-7b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 28.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Ministral 8B Instruct 2410",
   "model_id": "mistral/ministral-8b-instruct-2410",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 28.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OpenHermes 2 Mistral 7B",
   "model_id": "teknium/openhermes-2-mistral-7b",
   "provider": "teknium",
   "provider_display": "Teknium",
   "score": 28.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OpenHermes 2.5 Mistral 7B",
   "model_id": "teknium/openhermes-2-5-mistral-7b",
   "provider": "teknium",
   "provider_display": "Teknium",
   "score": 28.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-3.5-turbo",
   "model_id": "openai/gpt-3-5-turbo",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 28.3,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B 200K",
   "model_id": "01-ai/yi-6b-200k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 28.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 13B hf",
   "model_id": "meta/llama-2-13b-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 28.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 13B hf",
   "model_id": "nous-research/llama-2-13b-hf",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 28.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 1B Base",
   "model_id": "tii/falcon3-1b-base",
   "provider": "tii",
   "provider_display": "TII",
   "score": 27.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2B it",
   "model_id": "google/gemma-2b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 27.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 3B Instruct",
   "model_id": "meta/llama-3-2-3b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 27.9,
   "source": "llm-stats, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B Instruct v0.3",
   "model_id": "mistral/mistral-7b-instruct-v0-3",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 27.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OLMo 2 1124 7B",
   "model_id": "allen-ai/olmo-2-1124-7b",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 27.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 10B Base",
   "model_id": "tii/falcon3-10b-base",
   "provider": "tii",
   "provider_display": "TII",
   "score": 27.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 10B Instruct",
   "model_id": "tii/falcon3-10b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 27.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B Instruct v0.2",
   "model_id": "mistral/mistral-7b-instruct-v0-2",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 27.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 3 Llama 3.2 3B",
   "model_id": "nous-research/hermes-3-llama-3-2-3b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 27.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Infinity Instruct 3M 0625 Llama3 8B",
   "model_id": "baai/infinity-instruct-3m-0625-llama3-8b",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 27.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek llm 7B base",
   "model_id": "deepseek/deepseek-llm-7b-base",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 27.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 40B",
   "model_id": "tii/falcon-40b",
   "provider": "tii",
   "provider_display": "TII",
   "score": 27.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OpenHermes 13B",
   "model_id": "teknium/openhermes-13b",
   "provider": "teknium",
   "provider_display": "Teknium",
   "score": 27.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 3B Instruct",
   "model_id": "qwen/qwen2-5-3b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 27.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x7B",
   "model_id": "mistral/open-mixtral-8x7b",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 27.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x7B Instruct v0.1",
   "model_id": "mistral/mixtral-8x7b-instruct-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 27.2,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x7B v0.1",
   "model_id": "mistral/mixtral-8x7b-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 27.2,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "phi 2",
   "model_id": "microsoft/phi-2",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 27.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 11B",
   "model_id": "tii/falcon-11b",
   "provider": "tii",
   "provider_display": "TII",
   "score": 27.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "gemma 1.1 2B it",
   "model_id": "google/gemma-1-1-2b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 26.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B",
   "model_id": "01-ai/yi-6b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 26.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Jamba v0.1",
   "model_id": "ai21/jamba-v0-1",
   "provider": "ai21",
   "provider_display": "AI21 Labs",
   "score": 26.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 1B Instruct",
   "model_id": "meta/llama-3-2-1b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 26.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 3B",
   "model_id": "meta/llama-3-2-3b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 26.8,
   "source": "llm-stats, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 1B Instruct",
   "model_id": "tii/falcon3-1b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 26.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B hf",
   "model_id": "meta/llama-2-7b-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 26.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B hf",
   "model_id": "nous-research/llama-2-7b-hf",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 26.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek llm 7B chat",
   "model_id": "deepseek/deepseek-llm-7b-chat",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 26.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Math 1.5B",
   "model_id": "qwen/qwen2-5-math-1-5b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 26.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 70B chat hf",
   "model_id": "meta/llama-2-70b-chat-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 26.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2 2B",
   "model_id": "google/gemma-2-2b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 26.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes llama 2 7B",
   "model_id": "nous-research/nous-hermes-llama-2-7b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 26.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "OLMo 1B hf",
   "model_id": "allen-ai/olmo-1b-hf",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 26.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 1.5B Instruct",
   "model_id": "qwen/qwen2-1-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 26.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "flan t5 small",
   "model_id": "google/flan-t5-small",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 26.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "GPT JT 6B v1",
   "model_id": "together/gpt-jt-6b-v1",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 26.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OLMoE 1B 7B 0125",
   "model_id": "allen-ai/olmoe-1b-7b-0125",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 26.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OLMoE 1B 7B 0125 Instruct",
   "model_id": "allen-ai/olmoe-1b-7b-0125-instruct",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 26.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 0.5B Instruct",
   "model_id": "qwen/qwen2-5-0-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 25.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 7B Instruct GPTQ Int4",
   "model_id": "qwen/qwen2-5-coder-7b-instruct-gptq-int4",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 25.9,
   "source": "bigcode-leaderboard, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Qwen 1.5B",
   "model_id": "deepseek/deepseek-r1-distill-qwen-1-5b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 25.6,
   "source": "bigcode-leaderboard, provider-reports, open-llm-leaderboard, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 1.5B Instruct",
   "model_id": "qwen/qwen2-5-1-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 25.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2B",
   "model_id": "google/gemma-2b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 25.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "RedPajama INCITE 7B Base",
   "model_id": "together/redpajama-incite-7b-base",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 25.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "deepseek moe 16B base",
   "model_id": "deepseek/deepseek-moe-16b-base",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 25.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "DialoGPT medium",
   "model_id": "microsoft/dialogpt-medium",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 25.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B chat hf",
   "model_id": "meta/llama-2-7b-chat-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 25.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B chat hf",
   "model_id": "nous-research/llama-2-7b-chat-hf",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 25.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "RedPajama INCITE 7B Chat",
   "model_id": "together/redpajama-incite-7b-chat",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 25.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B 32K Instruct",
   "model_id": "together/llama-2-7b-32k-instruct",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 25.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Ministral 3B (latest)",
   "model_id": "mistral/ministral-3b-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 25.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "flan t5 large",
   "model_id": "google/flan-t5-large",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 25.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "RedPajama INCITE 7B Instruct",
   "model_id": "together/redpajama-incite-7b-instruct",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 25.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 40B instruct",
   "model_id": "tii/falcon-40b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 25.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "GPT NeoXT Chat Base 20B",
   "model_id": "together/gpt-neoxt-chat-base-20b",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 25.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "LLaMA 2 7B 32K",
   "model_id": "together/llama-2-7b-32k",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 25.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B Instruct v0.1",
   "model_id": "mistral/mistral-7b-instruct-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 25.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "stablelm 2 1 6B",
   "model_id": "stability/stablelm-2-1-6b",
   "provider": "stability",
   "provider_display": "Stability AI",
   "score": 24.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 7B instruct",
   "model_id": "tii/falcon-7b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 24.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "granite 3.0 1B a400m base",
   "model_id": "ibm/granite-3-0-1b-a400m-base",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 24.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OLMoE 1B 7B 0924",
   "model_id": "allen-ai/olmoe-1b-7b-0924",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 24.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 0.5B Instruct",
   "model_id": "qwen/qwen2-0-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 24.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 0.5B",
   "model_id": "qwen/qwen2-5-0-5b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 24.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Yi Coder 9B",
   "model_id": "01-ai/yi-coder-9b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 24.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Yi Coder 9B Chat",
   "model_id": "01-ai/yi-coder-9b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 24.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "flan t5 xl",
   "model_id": "google/flan-t5-xl",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 24.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 7B",
   "model_id": "tii/falcon-7b",
   "provider": "tii",
   "provider_display": "TII",
   "score": 24.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "RedPajama INCITE Chat 3B v1",
   "model_id": "together/redpajama-incite-chat-3b-v1",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 24.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "RedPajama INCITE Base 3B v1",
   "model_id": "together/redpajama-incite-base-3b-v1",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 24.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "mt5 small",
   "model_id": "google/mt5-small",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 24.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "granite 3.1 1B a400m instruct",
   "model_id": "ibm/granite-3-1-1b-a400m-instruct",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 24.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "mt5 base",
   "model_id": "google/mt5-base",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 23.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "stablelm zephyr 3B",
   "model_id": "stability/stablelm-zephyr-3b",
   "provider": "stability",
   "provider_display": "Stability AI",
   "score": 23.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "flan t5 base",
   "model_id": "google/flan-t5-base",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 23.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "stablelm 3B 4e1t",
   "model_id": "stability/stablelm-3b-4e1t",
   "provider": "stability",
   "provider_display": "Stability AI",
   "score": 23.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 13B chat hf",
   "model_id": "meta/llama-2-13b-chat-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 23.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "rwkv raven 14B",
   "model_id": "rwkv/rwkv-raven-14b",
   "provider": "rwkv",
   "provider_display": "RWKV Foundation",
   "score": 22.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 1B",
   "model_id": "meta/llama-3-2-1b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 22.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 1B",
   "model_id": "nous-research/llama-3-2-1b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 22.8,
   "source": "open-llm-leaderboard-v2"
  }
 ],
 "page": {
  "aliases": [
   "GPQA-Diamond"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "The dataset ships with a canary string and a no-public-sharing agreement, and the GitHub copy is password-gated, specifically to slow contamination. But the questions have circulated, gated, since November 2023, and the benchmark is heavily used for both training and evaluation, so some leakage over almost three years is plausible despite those measures.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC BY 4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 198,
   "size_note": "198 questions: the subset of the 448-question main GPQA set where both expert annotators answered correctly and most non-expert validators did not.\n",
   "splits": "single evaluation set, no train/test split",
   "url": "https://huggingface.co/datasets/Idavidrein/gpqa"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice F",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "gpqa_diamond_zeroshot",
   "opencompass": "",
   "other": "OpenAI's simple-evals package (github.com/openai/simple-evals, gpqa_eval.py) supplies the zero-shot chain-of-thought prompt most labs now use when reporting GPQA Diamond scores.\n"
  },
  "id": "gpqa_diamond",
  "last_updated": "",
  "leaderboard_url": "https://epoch.ai/benchmarks/gpqa-diamond",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "GPQA Diamond tests whether a model can answer graduate-level multiple-choice questions in biology, physics and chemistry that are deliberately built to resist internet lookup. Each question was written and validated by a PhD-level domain expert, then filtered so that expert annotators agreed on the answer while most skilled non-experts, given open web access and significant time, did not. It is a single-turn, text-only, English-language task with four answer options and one correct letter, meant to reward domain reasoning rather than search skill.\n",
  "metric": {
   "baseline_note": "OpenAI recruited PhD-level experts to answer the Diamond subset specifically and measured 69.7% accuracy, reported in OpenAI's o1 announcement and cited on Epoch AI's methodology page. The original GPQA paper separately reports 65% accuracy (74% excluding self-identified mistakes) for expert validators across its broader validation sample, not Diamond alone.\n",
   "direction": "higher_is_better",
   "human_baseline": 69.7,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 25,
   "unit": "%"
  },
  "name": "GPQA Diamond",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2311.12022",
   "title": "GPQA: A Graduate-Level Google-Proof Q&A Benchmark",
   "url": "https://arxiv.org/abs/2311.12022",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "David Rein",
    "Betty Li Hou",
    "Asa Cooper Stickland",
    "Jackson Petty",
    "Richard Yuanzhe Pang",
    "Julien Dirani",
    "Julian Michael",
    "Samuel R. Bowman"
   ],
   "org": "",
   "url": "https://github.com/idavidrein/gpqa"
  },
  "released": "2023-11",
  "repo_url": "https://github.com/idavidrein/gpqa",
  "saturation": {
   "as_of": "2025-12",
   "note": "OpenAI's own tracking shows the trajectory: GPT-4 scored 39% at the benchmark's November 2023 release against OpenAI's later-measured 69.7% PhD-expert baseline, o1 reached 77.3% in September 2024, and GPT-5.2 scored 92% by December 2025 per OpenAI's FrontierScience announcement. Epoch AI separately measured Grok 4 at 87% (plus or minus 2%) in July 2025. Scores are now well clear of the expert baseline and close to the 100% ceiling.\n",
   "status": "saturated",
   "top_score": 92.0
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "GPQA: A Graduate-Level Google-Proof Q&A Benchmark",
    "url": "https://arxiv.org/abs/2311.12022"
   },
   {
    "accessed": "2026-09-07",
    "title": "idavidrein/gpqa (README, LICENSE, run_baseline.py)",
    "url": "https://github.com/idavidrein/gpqa"
   },
   {
    "accessed": "2026-09-07",
    "title": "Idavidrein/gpqa dataset card",
    "url": "https://huggingface.co/datasets/Idavidrein/gpqa"
   },
   {
    "accessed": "2026-09-07",
    "title": "GPQA Diamond | Epoch AI",
    "url": "https://epoch.ai/benchmarks/gpqa-diamond"
   },
   {
    "accessed": "2026-09-07",
    "title": "Evaluating AI's ability to perform scientific research tasks | OpenAI",
    "url": "https://openai.com/index/frontierscience/"
   }
  ],
  "status": "active",
  "subcategory": "graduate-level science Q&A",
  "summary": "198 PhD-written multiple-choice science questions built to resist lookup, the hardest subset of GPQA.",
  "tags": [
   "science",
   "multiple-choice",
   "phd-level",
   "chain-of-thought"
  ],
  "task_format": "Four-option multiple-choice question in biology, physics or chemistry; the model returns a single letter answer, typically after chain-of-thought reasoning.\n"
 }
}