{
 "body": "\n## What it measures\n\nIFEval tests whether a model obeys explicit, checkable instructions layered onto an otherwise ordinary\nprompt: constraints like writing more than 400 words, mentioning a keyword at least three times, avoiding\ncommas entirely, or wrapping the answer in double quotation marks. Roughly 500 prompts each combine one or\nmore of 25 such instruction types, covering length, keyword use, formatting, punctuation and casing. Every\ninstruction is written so that a program, not a person or another model, can check whether it was\nfollowed.\n\n## How it is scored\n\nA model answers each prompt zero-shot, with no examples shown. A verification function checks the\nresponse against every instruction in the prompt mechanically \u2014 counting words, scanning for a keyword,\nchecking capitalisation \u2014 and returns a pass or fail per instruction. Four related numbers are reported:\nstrict and loose accuracy, each computed at the prompt level (every instruction in the prompt must pass\nfor the prompt to count) and at the instruction level (each instruction scored on its own, so a\nthree-instruction prompt with two passes contributes partial credit at this level even though it fails at\nthe prompt level). \"Loose\" scoring tolerates minor response variations, such as extra markdown wrapping,\nthat \"strict\" scoring does not.\n\n## Dataset and licence\n\nAbout 500 prompts, released under CC BY 4.0 and hosted on Hugging Face as `google/IFEval`. The\nlm-evaluation-harness implementation reads the dataset's \"train\" split, since no separate test split is\nprovided.\n\n## Who publishes it\n\nIFEval comes from Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny\nZhou and Le Hou at Google Research, first posted to arXiv in November 2023. The reference dataset and\nscoring code are maintained in Google's `google-research` GitHub monorepo.\n\n## Lineage\n\nThis entry has no predecessor, successor or variant catalogued elsewhere in this repository; it is scored\nas a standalone benchmark by every model card that reports it.\n\n## Saturation and contamination\n\nNo primary leaderboard was opened for this page, so a current top score cannot be stated here with a\nsource behind it; this repository's own model cards show several current models scoring well above 90%,\nwhich is at least suggestive of a benchmark under ceiling pressure, but that pattern was not checked\nagainst an external leaderboard in this pass. Contamination risk is medium: the prompt set and its\nverification logic are fully public and have been in wide use since late 2023, so specific prompts are\nplausibly present in newer pretraining data, though the mechanical, non-answer-matched scoring means\nmemorising a specific correct answer is less directly rewarded than on a fixed-answer benchmark.\n\n## How to run it\n\nThe paper's own repository ships reference scoring code. The most common path today is\nlm-evaluation-harness's `ifeval` task, which generates greedily (no sampling) with a 1,280-token cap and\ncomputes all four accuracy variants automatically; it also underlies several third-party leaderboard\nsuites built on that harness. It was not confirmed in the HELM, OpenCompass or BIG-bench task lists.\n\n## Reading the numbers\n\nA high IFEval score shows a model reliably obeys explicit, literal formatting and content constraints \u2014\nuseful for anyone building on top of an API where downstream code parses the output. It says nothing about\nwhether the content of the response is correct, helpful, or well-reasoned, only whether its surface form\ncomplied with what was asked; a model can write a wrong or unhelpful answer that nonetheless satisfies\nevery instruction. Because strict and loose, and prompt-level and instruction-level, accuracy can diverge,\ncheck which of the four numbers is being reported before comparing scores across sources.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "ifeval",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.1",
   "model_id": "openai/gpt-5-1",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 92.5,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.1 Chat",
   "model_id": "openai/gpt-5-1-chat-latest",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 92.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.1 Codex",
   "model_id": "openai/gpt-5-1-codex",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 92.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.1 Codex Max",
   "model_id": "openai/gpt-5-1-codex-max",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 92.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.1 Codex mini",
   "model_id": "openai/gpt-5-1-codex-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 92.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Mythos Preview",
   "model_id": "anthropic/claude-mythos-preview",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 92.1,
   "source": "anthropic-system-card"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4",
   "model_id": "anthropic/claude-opus-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 92.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.6",
   "model_id": "anthropic/claude-opus-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 92.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, anthropic-system-card-mythos"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o3-pro",
   "model_id": "openai/o3-pro",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 92,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5",
   "model_id": "openai/gpt-5",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 91.8,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5 Chat (latest)",
   "model_id": "openai/gpt-5-chat-latest",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 91.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5 Pro",
   "model_id": "openai/gpt-5-pro",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 91.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5-Codex",
   "model_id": "openai/gpt-5-codex",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 91.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.2",
   "model_id": "openai/gpt-5-2",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 91.8,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.2 Chat",
   "model_id": "openai/gpt-5-2-chat-latest",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 91.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.2 Codex",
   "model_id": "openai/gpt-5-2-codex",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 91.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.2 Pro",
   "model_id": "openai/gpt-5-2-pro",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 91.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.3 Chat (latest)",
   "model_id": "openai/gpt-5-3-chat-latest",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 91.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.3 Codex",
   "model_id": "openai/gpt-5-3-codex",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 91.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.3 Codex Spark",
   "model_id": "openai/gpt-5-3-codex-spark",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 91.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.4",
   "model_id": "openai/gpt-5-4",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 91.8,
   "source": "lmarena.ai, provider-reports, anthropic-system-card-mythos, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.4 mini",
   "model_id": "openai/gpt-5-4-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 91.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.4 nano",
   "model_id": "openai/gpt-5-4-nano",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 91.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.4 Pro",
   "model_id": "openai/gpt-5-4-pro",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 91.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.5",
   "model_id": "anthropic/claude-opus-4-5-20251101",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 91.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.5 (latest)",
   "model_id": "anthropic/claude-opus-4-5",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 91.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.1",
   "model_id": "anthropic/claude-opus-4-1-20250805",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 91.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.1 (latest)",
   "model_id": "anthropic/claude-opus-4-1",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 91.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro",
   "model_id": "google/gemini-2-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 91.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 05-06",
   "model_id": "google/gemini-2-5-pro-preview-05-06",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 91.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 06-05",
   "model_id": "google/gemini-2-5-pro-preview-06-05",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 91.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview TTS",
   "model_id": "google/gemini-2-5-pro-preview-tts",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 91.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o3",
   "model_id": "openai/o3",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 91.0,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o3-deep-research",
   "model_id": "openai/o3-deep-research",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 91,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4",
   "model_id": "anthropic/claude-sonnet-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 90.4,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.6",
   "model_id": "anthropic/claude-sonnet-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 90.4,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4",
   "model_id": "xai/grok-4",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 90.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4 Fast",
   "model_id": "xai/grok-4-fast",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 90.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4 Fast (Non-Reasoning)",
   "model_id": "xai/grok-4-fast-non-reasoning",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 90.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4.1 Fast",
   "model_id": "xai/grok-4-1-fast",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 90.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4.1 Fast (Non-Reasoning)",
   "model_id": "xai/grok-4-1-fast-non-reasoning",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 90.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4.20 (Non-Reasoning)",
   "model_id": "xai/grok-4-20-0309-non-reasoning",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 90.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4.20 (Reasoning)",
   "model_id": "xai/grok-4-20-0309-reasoning",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 90.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 4.20 Multi-Agent",
   "model_id": "xai/grok-4-20-multi-agent-0309",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 90.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4 (latest)",
   "model_id": "anthropic/claude-opus-4-0",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 89.7,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1",
   "model_id": "openai/gpt-4-1",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 89.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, domain-evals preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.5",
   "model_id": "anthropic/claude-sonnet-4-5-20250929",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 89.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.5 (latest)",
   "model_id": "anthropic/claude-sonnet-4-5",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 89.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o4-mini",
   "model_id": "openai/o4-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 89.0,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o4-mini-deep-research",
   "model_id": "openai/o4-mini-deep-research",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 89.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o1-pro",
   "model_id": "openai/o1-pro",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 88.5,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 235B-A22B",
   "model_id": "qwen/qwen3-235b-a22b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 88.5,
   "source": "lmarena.ai, provider-reports, preference-evals, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 0528",
   "model_id": "deepseek/deepseek-r1-0528",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 88.0,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 0528 NVFP4 v2",
   "model_id": "nvidia/deepseek-r1-0528-nvfp4-v2",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 88.0,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5 Mini",
   "model_id": "openai/gpt-5-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 88.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3-Coder 480B-A35B Instruct",
   "model_id": "qwen/qwen3-coder-480b-a35b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 88.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash",
   "model_id": "google/gemini-2-5-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.5,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Image",
   "model_id": "google/gemini-2-5-flash-image",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Image (Preview)",
   "model_id": "google/gemini-2-5-flash-image-preview",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Lite",
   "model_id": "google/gemini-2-5-flash-lite",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Lite Preview 06-17",
   "model_id": "google/gemini-2-5-flash-lite-preview-06-17",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Lite Preview 09-25",
   "model_id": "google/gemini-2-5-flash-lite-preview-09-2025",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Preview 04-17",
   "model_id": "google/gemini-2-5-flash-preview-04-17",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Preview 05-20",
   "model_id": "google/gemini-2-5-flash-preview-05-20",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Preview 09-25",
   "model_id": "google/gemini-2-5-flash-preview-09-2025",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Preview TTS",
   "model_id": "google/gemini-2-5-flash-preview-tts",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4 (latest)",
   "model_id": "anthropic/claude-sonnet-4-0",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 87.2,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.7",
   "model_id": "anthropic/claude-3-7-sonnet-20250219",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 87.0,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3.2",
   "model_id": "deepseek/deepseek-v3-2",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 87.0,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3.2 Exp",
   "model_id": "deepseek/deepseek-v3-2-exp",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 87.0,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1",
   "model_id": "deepseek/deepseek-r1",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 86.5,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek Reasoner",
   "model_id": "deepseek/deepseek-reasoner",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 86.5,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o3-mini",
   "model_id": "openai/o3-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 86.5,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o1",
   "model_id": "openai/o1",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 86.0,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o1-preview",
   "model_id": "openai/o1-preview",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 86.0,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3.1",
   "model_id": "deepseek/deepseek-v3-1",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 85.5,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5",
   "model_id": "anthropic/claude-3-5-sonnet-20240620",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 85.4,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5 v2",
   "model_id": "anthropic/claude-3-5-sonnet-20241022",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 85.4,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 4.5",
   "model_id": "anthropic/claude-haiku-4-5-20251001",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 85.0,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 4.5 (latest)",
   "model_id": "anthropic/claude-haiku-4-5",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 85.0,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1 mini",
   "model_id": "openai/gpt-4-1-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 85.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3",
   "model_id": "xai/grok-3",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 85.0,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3 Fast",
   "model_id": "xai/grok-3-fast",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 85.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3 Fast Latest",
   "model_id": "xai/grok-3-fast-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 85.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3 Latest",
   "model_id": "xai/grok-3-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 85.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Magistral Medium (latest)",
   "model_id": "mistral/magistral-medium-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 85.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o",
   "model_id": "openai/gpt-4o",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 84.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-05-13)",
   "model_id": "openai/gpt-4o-2024-05-13",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 84.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-08-06)",
   "model_id": "openai/gpt-4o-2024-08-06",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 84.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-11-20)",
   "model_id": "openai/gpt-4o-2024-11-20",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 84.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek Chat",
   "model_id": "deepseek/deepseek-chat",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 84.0,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V2",
   "model_id": "deepseek/deepseek-v2",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 84,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V2 Lite",
   "model_id": "deepseek/deepseek-v2-lite",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 84,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V2 Lite Chat",
   "model_id": "deepseek/deepseek-v2-lite-chat",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 84,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3",
   "model_id": "deepseek/deepseek-v3",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 84.0,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, domain-evals open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3 0324",
   "model_id": "deepseek/deepseek-v3-0324",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 84.0,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.0 Flash",
   "model_id": "google/gemini-2-0-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 84.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 32B",
   "model_id": "qwen/qwen3-32b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 84.0,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 32B AWQ",
   "model_id": "qwen/qwen3-32b-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 84.0,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 32B NVFP4",
   "model_id": "nvidia/qwen3-32b-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 84.0,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 31B",
   "model_id": "google/gemma-4-31b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 83.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 31B it",
   "model_id": "google/gemma-4-31b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 83.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 31B it GGUF",
   "model_id": "unsloth/gemma-4-31b-it-gguf",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 83.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 31B IT NVFP4",
   "model_id": "nvidia/gemma-4-31b-it-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 83.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Maverick 17B 128E Instruct",
   "model_id": "meta/llama-4-maverick-17b-128e-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 83.0,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-4-Maverick-17B-128E-Instruct-FP8",
   "model_id": "meta/llama-4-maverick-17b-128e-instruct-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 83.0,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5 Nano",
   "model_id": "openai/gpt-5-nano",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Pro",
   "model_id": "google/gemini-1-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 82.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4 Turbo",
   "model_id": "openai/gpt-4-turbo",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 82.0,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large (latest)",
   "model_id": "mistral/mistral-large-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 82.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large 2.1",
   "model_id": "mistral/mistral-large-2411",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 82.0,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals,, domain-evals open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large 3",
   "model_id": "mistral/mistral-large-2512",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 82.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o1-mini",
   "model_id": "openai/o1-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 82.0,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 72B Instruct",
   "model_id": "qwen/qwen2-5-72b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 82.0,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 26B",
   "model_id": "google/gemma-4-26b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 81.5,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 26B A4B it",
   "model_id": "google/gemma-4-26b-a4b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 81.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 26B A4B it GGUF",
   "model_id": "unsloth/gemma-4-26b-a4b-it-gguf",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 81.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 3",
   "model_id": "anthropic/claude-3-opus-20240229",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 81.0,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B Instruct",
   "model_id": "meta/meta-llama-3-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 81.0,
   "source": "llm-stats, open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B Instruct",
   "model_id": "nous-research/meta-llama-3-70b-instruct",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 81.0,
   "source": "llm-stats, open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3.5",
   "model_id": "anthropic/claude-3-5-haiku-20241022",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 80.5,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3.5 (latest)",
   "model_id": "anthropic/claude-3-5-haiku-latest",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 80.5,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o mini",
   "model_id": "openai/gpt-4o-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 80.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3 Mini",
   "model_id": "xai/grok-3-mini",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 80.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3 Mini Fast",
   "model_id": "xai/grok-3-mini-fast",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 80.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3 Mini Fast Latest",
   "model_id": "xai/grok-3-mini-fast-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 80.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 3 Mini Latest",
   "model_id": "xai/grok-3-mini-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 80.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 405B",
   "model_id": "meta/llama-3-1-405b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 80.0,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 405B FP8",
   "model_id": "meta/llama-3-1-405b-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 80.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 405B Instruct",
   "model_id": "meta/llama-3-1-405b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 80.0,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 405B Instruct FP8",
   "model_id": "meta/llama-3-1-405b-instruct-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 80.0,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 30B A3B Instruct 2507",
   "model_id": "qwen/qwen3-30b-a3b-instruct-2507",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 80.0,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 30B A3B NVFP4",
   "model_id": "nvidia/qwen3-30b-a3b-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 80.0,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 30B-A3B",
   "model_id": "qwen/qwen3-30b-a3b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 80.0,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "QwQ Plus",
   "model_id": "qwen/qwq-plus",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 80,
   "source": "provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 27B",
   "model_id": "google/gemma-3-27b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 79.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2,, domain-evals llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Scout 17B 16E",
   "model_id": "meta/llama-4-scout-17b-16e",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 79.0,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Scout 17B 16E Instruct",
   "model_id": "meta/llama-4-scout-17b-16e-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 79.0,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-4-Scout-17B-16E-Instruct-FP8",
   "model_id": "meta/llama-4-scout-17b-16e-instruct-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 79.0,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.0 Flash Lite",
   "model_id": "google/gemini-2-0-flash-lite",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 78.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.3 70B Instruct NVFP4",
   "model_id": "nvidia/llama-3-3-70b-instruct-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 78.5,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-3.3-70B-Instruct",
   "model_id": "meta/llama-3-3-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 78.5,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Command A",
   "model_id": "cohere/command-a-03-2025",
   "provider": "cohere",
   "provider_display": "Cohere",
   "score": 78,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Command A Reasoning",
   "model_id": "cohere/command-a-reasoning-08-2025",
   "provider": "cohere",
   "provider_display": "Cohere",
   "score": 78,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Llama 70B",
   "model_id": "deepseek/deepseek-r1-distill-llama-70b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 78.0,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1 nano",
   "model_id": "openai/gpt-4-1-nano",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 78.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2",
   "model_id": "xai/grok-2",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 78,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 (1212)",
   "model_id": "xai/grok-2-1212",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 78,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Latest",
   "model_id": "xai/grok-2-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 78,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Vision",
   "model_id": "xai/grok-2-vision",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 78,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Vision (1212)",
   "model_id": "xai/grok-2-vision-1212",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 78,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Vision Latest",
   "model_id": "xai/grok-2-vision-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 78,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Magistral Small",
   "model_id": "mistral/magistral-small",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 78.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Magistral Small 2506",
   "model_id": "mistral/magistral-small-2506",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 78.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "phi 4",
   "model_id": "microsoft/phi-4",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 78.0,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2,, domain-evals llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Phi 4 multimodal instruct",
   "model_id": "microsoft/phi-4-multimodal-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 78,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 32B Instruct",
   "model_id": "qwen/qwen2-5-32b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 78,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 32B Instruct AWQ",
   "model_id": "qwen/qwen2-5-32b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 78,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x22B",
   "model_id": "mistral/open-mixtral-8x22b",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 77.8,
   "source": "open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x22B Instruct v0.1",
   "model_id": "mistral/mixtral-8x22b-instruct-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 77.8,
   "source": "open-llm-leaderboard-v2, llm-stats, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B",
   "model_id": "01-ai/yi-1-5-34b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 77.3,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B Chat",
   "model_id": "01-ai/yi-1-5-34b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 77.3,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Flash",
   "model_id": "google/gemini-1-5-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 77.0,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Flash-8B",
   "model_id": "google/gemini-1-5-flash-8b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 77.0,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 14B",
   "model_id": "qwen/qwen3-14b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 77,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 14B AWQ",
   "model_id": "qwen/qwen3-14b-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 77,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 14B NVFP4",
   "model_id": "nvidia/qwen3-14b-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 77,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4",
   "model_id": "openai/gpt-4",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 76.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 7B Instruct",
   "model_id": "tii/falcon3-7b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 76.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Qwen 32B",
   "model_id": "deepseek/deepseek-r1-distill-qwen-32b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 76.0,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 70B",
   "model_id": "meta/llama-3-1-70b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 76.0,
   "source": "lmarena.ai, provider-reports, safety-evals, domain-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 70B Instruct",
   "model_id": "meta/llama-3-1-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 76.0,
   "source": "lmarena.ai, provider-reports, safety-evals, domain-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 32B Instruct",
   "model_id": "qwen/qwen2-5-coder-32b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 76.0,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 32B Instruct AWQ",
   "model_id": "qwen/qwen2-5-coder-32b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 76.0,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3",
   "model_id": "anthropic/claude-3-sonnet-20240229",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 75.5,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Nemo",
   "model_id": "mistral/mistral-nemo",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 75.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Nemo Base 2407",
   "model_id": "mistral/mistral-nemo-base-2407",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 75.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Nemo Instruct 2407",
   "model_id": "mistral/mistral-nemo-instruct-2407",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 75.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Command R+",
   "model_id": "cohere/command-r-plus-08-2024",
   "provider": "cohere",
   "provider_display": "Cohere",
   "score": 75.0,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2 27B it",
   "model_id": "google/gemma-2-27b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 75.0,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Codestral (latest)",
   "model_id": "mistral/codestral-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 74.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 14B Instruct",
   "model_id": "qwen/qwen2-5-14b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 74,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 14B Instruct AWQ",
   "model_id": "qwen/qwen2-5-14b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 74,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 3B Instruct",
   "model_id": "meta/llama-3-2-3b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 73.9,
   "source": "llm-stats, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 12B",
   "model_id": "google/gemma-3-12b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 73.0,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 8B",
   "model_id": "qwen/qwen3-8b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 73,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 8B AWQ",
   "model_id": "qwen/qwen3-8b-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 73,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 8B Base",
   "model_id": "qwen/qwen3-8b-base",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 73,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OLMo 2 1124 7B",
   "model_id": "allen-ai/olmo-2-1124-7b",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 72.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3.5 mini instruct",
   "model_id": "microsoft/phi-3-5-mini-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 72.4,
   "source": "open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "granite 3.1 8B instruct",
   "model_id": "ibm/granite-3-1-8b-instruct",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 72.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3",
   "model_id": "anthropic/claude-3-haiku-20240307",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 72.0,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Qwen 14B",
   "model_id": "deepseek/deepseek-r1-distill-qwen-14b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 72.0,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Phi 4 mini instruct",
   "model_id": "microsoft/phi-4-mini-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 72.0,
   "source": "lmarena.ai, provider-reports, preference-evals, llm-stats, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 Mamba 7B Instruct",
   "model_id": "tii/falcon3-mamba-7b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 71.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Command R",
   "model_id": "cohere/command-r-08-2024",
   "provider": "cohere",
   "provider_display": "Cohere",
   "score": 70,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 3B Instruct",
   "model_id": "tii/falcon3-3b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 69.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x7B",
   "model_id": "mistral/open-mixtral-8x7b",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 69.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x7B Instruct v0.1",
   "model_id": "mistral/mixtral-8x7b-instruct-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 69.5,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x7B v0.1",
   "model_id": "mistral/mixtral-8x7b-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 69.5,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 10B Base",
   "model_id": "tii/falcon3-10b-base",
   "provider": "tii",
   "provider_display": "TII",
   "score": 69.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 10B Instruct",
   "model_id": "tii/falcon3-10b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 69.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 14B Instruct",
   "model_id": "qwen/qwen2-5-coder-14b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 69.1,
   "source": "bigcode-leaderboard, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2 9B",
   "model_id": "google/gemma-2-9b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 68.0,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2 9B it",
   "model_id": "google/gemma-2-9b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 68.0,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 7B",
   "model_id": "qwen/qwen2-5-7b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 68.0,
   "source": "provider-reports, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 7B Instruct",
   "model_id": "qwen/qwen2-5-7b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 68,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 7B Instruct AWQ",
   "model_id": "qwen/qwen2-5-7b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 68,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OLMoE 1B 7B 0125",
   "model_id": "allen-ai/olmoe-1b-7b-0125",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 67.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OLMoE 1B 7B 0125 Instruct",
   "model_id": "allen-ai/olmoe-1b-7b-0125-instruct",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 67.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Qwen 7B",
   "model_id": "deepseek/deepseek-r1-distill-qwen-7b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 66,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 4B",
   "model_id": "qwen/qwen3-4b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 66,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 4B Base",
   "model_id": "qwen/qwen3-4b-base",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 66,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 4B Instruct 2507",
   "model_id": "qwen/qwen3-4b-instruct-2507",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 66,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 4B Instruct 2507 FP8",
   "model_id": "qwen/qwen3-4b-instruct-2507-fp8",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 66,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 2 Theta Llama 3 8B",
   "model_id": "nous-research/hermes-2-theta-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 65.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "glm 4 9B chat hf",
   "model_id": "zhipu/glm-4-9b-chat-hf",
   "provider": "zhipu",
   "provider_display": "Zhipu AI",
   "score": 65.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 3B Instruct",
   "model_id": "qwen/qwen2-5-3b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 64.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Llama 8B",
   "model_id": "deepseek/deepseek-r1-distill-llama-8b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 64,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "granite 3.1 2B instruct",
   "model_id": "ibm/granite-3-1-2b-instruct",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 62.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 4B",
   "model_id": "google/gemma-3-4b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 62.0,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 3 4B pt",
   "model_id": "google/gemma-3-4b-pt",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 62.0,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B",
   "model_id": "meta/llama-3-1-8b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 62,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct",
   "model_id": "meta/llama-3-1-8b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 62,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct",
   "model_id": "unsloth/llama-3-1-8b-instruct",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 62,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct FP8",
   "model_id": "nvidia/llama-3-1-8b-instruct-fp8",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 62,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct NVFP4",
   "model_id": "nvidia/llama-3-1-8b-instruct-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 62,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 3 Llama 3.1 8B",
   "model_id": "nous-research/hermes-3-llama-3-1-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 61.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 3 Llama 3.1 8B GGUF",
   "model_id": "nous-research/hermes-3-llama-3-1-8b-gguf",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 61.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 7B Instruct",
   "model_id": "qwen/qwen2-5-coder-7b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 61.5,
   "source": "bigcode-leaderboard, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Infinity Instruct 7M Gen Llama3 1 8B",
   "model_id": "baai/infinity-instruct-7m-gen-llama3-1-8b",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 61.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-3.5-turbo",
   "model_id": "openai/gpt-3-5-turbo",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 60.5,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Infinity Instruct 3M 0625 Llama3 8B",
   "model_id": "baai/infinity-instruct-3m-0625-llama3-8b",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 60.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B Chat",
   "model_id": "01-ai/yi-1-5-9b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 60.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3 mini 128K instruct",
   "model_id": "microsoft/phi-3-mini-128k-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 59.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Ministral 8B Instruct 2410",
   "model_id": "mistral/ministral-8b-instruct-2410",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 59.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes 2 Mixtral 8x7B DPO",
   "model_id": "nous-research/nous-hermes-2-mixtral-8x7b-dpo",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 59.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 1B Instruct",
   "model_id": "meta/llama-3-2-1b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 58.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OpenHermes 2.5 Mistral 7B",
   "model_id": "teknium/openhermes-2-5-mistral-7b",
   "provider": "teknium",
   "provider_display": "Teknium",
   "score": 55.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 1B Instruct",
   "model_id": "tii/falcon3-1b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 55.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B Instruct v0.2",
   "model_id": "mistral/mistral-7b-instruct-v0-2",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 55.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B Instruct v0.3",
   "model_id": "mistral/mistral-7b-instruct-v0-3",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 54.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3 mini 4K instruct",
   "model_id": "microsoft/phi-3-mini-4k-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 54.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 2 Pro Llama 3 8B",
   "model_id": "nous-research/hermes-2-pro-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 53.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "granite 3.0 8B instruct",
   "model_id": "ibm/granite-3-0-8b-instruct",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 53.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OpenHermes 2 Mistral 7B",
   "model_id": "teknium/openhermes-2-mistral-7b",
   "provider": "teknium",
   "provider_display": "Teknium",
   "score": 52.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes 2 SOLAR 10.7B",
   "model_id": "nous-research/nous-hermes-2-solar-10-7b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 52.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Infinity Instruct 3M 0625 Yi 1.5 9B",
   "model_id": "baai/infinity-instruct-3m-0625-yi-1-5-9b",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 51.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 6B Chat",
   "model_id": "01-ai/yi-1-5-6b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 51.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 70B chat hf",
   "model_id": "meta/llama-2-70b-chat-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 49.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Yi Coder 9B",
   "model_id": "01-ai/yi-coder-9b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 48.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Yi Coder 9B Chat",
   "model_id": "01-ai/yi-coder-9b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 48.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B Instruct",
   "model_id": "meta/meta-llama-3-8b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 47.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B Instruct",
   "model_id": "nous-research/meta-llama-3-8b-instruct",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 47.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 34B Chat",
   "model_id": "01-ai/yi-34b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 47.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "granite 3.1 1B a400m instruct",
   "model_id": "ibm/granite-3-1-1b-a400m-instruct",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 46.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 VL 7B Instruct",
   "model_id": "qwen/qwen2-vl-7b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 46.0,
   "source": "open-llm-leaderboard-v2, multimodal-evals"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 VL 7B Instruct AWQ",
   "model_id": "qwen/qwen2-vl-7b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 46.0,
   "source": "open-llm-leaderboard-v2, multimodal-evals"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B Chat 16K",
   "model_id": "01-ai/yi-1-5-34b-chat-16k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 45.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B Instruct v0.1",
   "model_id": "mistral/mistral-7b-instruct-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 44.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 1.5B Instruct",
   "model_id": "qwen/qwen2-5-1-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 44.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B Chat 16K",
   "model_id": "01-ai/yi-1-5-9b-chat-16k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 42.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek llm 7B chat",
   "model_id": "deepseek/deepseek-llm-7b-chat",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 41.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B chat hf",
   "model_id": "meta/llama-2-7b-chat-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 39.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B chat hf",
   "model_id": "nous-research/llama-2-7b-chat-hf",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 39.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 13B chat hf",
   "model_id": "meta/llama-2-13b-chat-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 39.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 7B it",
   "model_id": "google/gemma-7b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 38.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 3 Llama 3.2 3B",
   "model_id": "nous-research/hermes-3-llama-3-2-3b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 38.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "stablelm zephyr 3B",
   "model_id": "stability/stablelm-zephyr-3b",
   "provider": "stability",
   "provider_display": "Stability AI",
   "score": 36.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Qwen 1.5B",
   "model_id": "deepseek/deepseek-r1-distill-qwen-1-5b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 34.6,
   "source": "bigcode-leaderboard, provider-reports, open-llm-leaderboard, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 7B Instruct GPTQ Int4",
   "model_id": "qwen/qwen2-5-coder-7b-instruct-gptq-int4",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 34.5,
   "source": "bigcode-leaderboard, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 7B Base",
   "model_id": "tii/falcon3-7b-base",
   "provider": "tii",
   "provider_display": "TII",
   "score": 34.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B Chat",
   "model_id": "01-ai/yi-6b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 34.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B Chat 4bits",
   "model_id": "01-ai/yi-6b-chat-4bits",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 34.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 1.5B Instruct",
   "model_id": "qwen/qwen2-1-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 33.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "falcon mamba 7B",
   "model_id": "tii/falcon-mamba-7b",
   "provider": "tii",
   "provider_display": "TII",
   "score": 33.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "falcon mamba 7B instruct",
   "model_id": "tii/falcon-mamba-7b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 33.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "falcon mamba 7B instruct Q4 K M GGUF",
   "model_id": "tii/falcon-mamba-7b-instruct-q4-k-m-gguf",
   "provider": "tii",
   "provider_display": "TII",
   "score": 33.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B",
   "model_id": "cerebras/llama3-1-8b",
   "provider": "cerebras",
   "provider_display": "Cerebras",
   "score": 33.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 11B",
   "model_id": "tii/falcon-11b",
   "provider": "tii",
   "provider_display": "TII",
   "score": 32.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 0.5B Instruct",
   "model_id": "qwen/qwen2-5-0-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 31.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B 32K",
   "model_id": "01-ai/yi-1-5-34b-32k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 31.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "gemma 1.1 2B it",
   "model_id": "google/gemma-1-1-2b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 30.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 34B",
   "model_id": "01-ai/yi-34b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 30.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B",
   "model_id": "01-ai/yi-1-5-9b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 29.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B",
   "model_id": "01-ai/yi-6b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 28.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 3B Base",
   "model_id": "tii/falcon3-3b-base",
   "provider": "tii",
   "provider_display": "TII",
   "score": 27.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "phi 2",
   "model_id": "microsoft/phi-2",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 27.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 9B",
   "model_id": "01-ai/yi-9b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 27.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2B it",
   "model_id": "google/gemma-2b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 26.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OpenHermes 13B",
   "model_id": "teknium/openhermes-13b",
   "provider": "teknium",
   "provider_display": "Teknium",
   "score": 26.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 1.5B",
   "model_id": "qwen/qwen2-5-1-5b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 26.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 1.5B Instruct AWQ",
   "model_id": "qwen/qwen2-5-1-5b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 26.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 6B",
   "model_id": "01-ai/yi-1-5-6b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 26.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B hf",
   "model_id": "meta/llama-2-7b-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 25.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B hf",
   "model_id": "nous-research/llama-2-7b-hf",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 25.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 40B",
   "model_id": "tii/falcon-40b",
   "provider": "tii",
   "provider_display": "TII",
   "score": 25.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 13B hf",
   "model_id": "meta/llama-2-13b-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 24.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 13B hf",
   "model_id": "nous-research/llama-2-13b-hf",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 24.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "deepseek moe 16B base",
   "model_id": "deepseek/deepseek-moe-16b-base",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 24.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 40B instruct",
   "model_id": "tii/falcon-40b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 24.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 1B Base",
   "model_id": "tii/falcon3-1b-base",
   "provider": "tii",
   "provider_display": "TII",
   "score": 24.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 70B hf",
   "model_id": "meta/llama-2-70b-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 24.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "granite 3.0 1B a400m base",
   "model_id": "ibm/granite-3-0-1b-a400m-base",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 24.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B v0.1",
   "model_id": "mistral/mistral-7b-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 23.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Yi 9B 200K",
   "model_id": "01-ai/yi-9b-200k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 23.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B 32K",
   "model_id": "01-ai/yi-1-5-9b-32k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 23.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "RedPajama INCITE Base 3B v1",
   "model_id": "together/redpajama-incite-base-3b-v1",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 22.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B v0.3",
   "model_id": "mistral/mistral-7b-v0-3",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 22.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "mistral 7B v0.3 bnb 4bit",
   "model_id": "unsloth/mistral-7b-v0-3-bnb-4bit",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 22.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 0.5B Instruct",
   "model_id": "qwen/qwen2-0-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 22.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "flan t5 xl",
   "model_id": "google/flan-t5-xl",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 22.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "flan t5 large",
   "model_id": "google/flan-t5-large",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 22.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "stablelm 3B 4e1t",
   "model_id": "stability/stablelm-3b-4e1t",
   "provider": "stability",
   "provider_display": "Stability AI",
   "score": 22.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek llm 7B base",
   "model_id": "deepseek/deepseek-llm-7b-base",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 21.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "OLMo 1B hf",
   "model_id": "allen-ai/olmo-1b-hf",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 21.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OLMoE 1B 7B 0924",
   "model_id": "allen-ai/olmoe-1b-7b-0924",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 21.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B 32K Instruct",
   "model_id": "together/llama-2-7b-32k-instruct",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 21.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "RedPajama INCITE 7B Base",
   "model_id": "together/redpajama-incite-7b-base",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 20.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "GPT JT 6B v1",
   "model_id": "together/gpt-jt-6b-v1",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 20.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "RedPajama INCITE 7B Instruct",
   "model_id": "together/redpajama-incite-7b-instruct",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 20.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2B",
   "model_id": "google/gemma-2b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 20.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Jamba v0.1",
   "model_id": "ai21/jamba-v0-1",
   "provider": "ai21",
   "provider_display": "AI21 Labs",
   "score": 20.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2 2B",
   "model_id": "google/gemma-2-2b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 20.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 7B instruct",
   "model_id": "tii/falcon-7b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 19.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "flan t5 base",
   "model_id": "google/flan-t5-base",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 18.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "LLaMA 2 7B 32K",
   "model_id": "together/llama-2-7b-32k",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 18.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Math 1.5B",
   "model_id": "qwen/qwen2-5-math-1-5b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 18.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "GPT NeoXT Chat Base 20B",
   "model_id": "together/gpt-neoxt-chat-base-20b",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 18.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 7B",
   "model_id": "tii/falcon-7b",
   "provider": "tii",
   "provider_display": "TII",
   "score": 18.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes llama 2 7B",
   "model_id": "nous-research/nous-hermes-llama-2-7b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 17.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "mt5 small",
   "model_id": "google/mt5-small",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 17.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "mt5 base",
   "model_id": "google/mt5-base",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 16.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "RedPajama INCITE Chat 3B v1",
   "model_id": "together/redpajama-incite-chat-3b-v1",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 16.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 0.5B",
   "model_id": "qwen/qwen2-5-0-5b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 16.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B",
   "model_id": "meta/meta-llama-3-70b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 16.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "RedPajama INCITE 7B Chat",
   "model_id": "together/redpajama-incite-7b-chat",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 15.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 34B 200K",
   "model_id": "01-ai/yi-34b-200k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 15.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "flan t5 small",
   "model_id": "google/flan-t5-small",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 15.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "DialoGPT medium",
   "model_id": "microsoft/dialogpt-medium",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 14.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 1B",
   "model_id": "meta/llama-3-2-1b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 14.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 1B",
   "model_id": "nous-research/llama-3-2-1b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 14.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B",
   "model_id": "meta/meta-llama-3-8b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 14.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B",
   "model_id": "nous-research/meta-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 14.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "glm 4 9B",
   "model_id": "zhipu/glm-4-9b",
   "provider": "zhipu",
   "provider_display": "Zhipu AI",
   "score": 14.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "GLM 4 9B 0414",
   "model_id": "zhipu/glm-4-9b-0414",
   "provider": "zhipu",
   "provider_display": "Zhipu AI",
   "score": 14.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Ministral 3B (latest)",
   "model_id": "mistral/ministral-3b-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 13.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 3B",
   "model_id": "meta/llama-3-2-3b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 13.4,
   "source": "llm-stats, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2 2B it",
   "model_id": "google/gemma-2-2b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 12.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "stablelm 2 1 6B",
   "model_id": "stability/stablelm-2-1-6b",
   "provider": "stability",
   "provider_display": "Stability AI",
   "score": 11.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B 200K",
   "model_id": "01-ai/yi-6b-200k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 8.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "rwkv raven 14B",
   "model_id": "rwkv/rwkv-raven-14b",
   "provider": "rwkv",
   "provider_display": "RWKV Foundation",
   "score": 7.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "glm 4 9B chat",
   "model_id": "zhipu/glm-4-9b-chat",
   "provider": "zhipu",
   "provider_display": "Zhipu AI",
   "score": 0.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "glm 4 9B chat 1M",
   "model_id": "zhipu/glm-4-9b-chat-1m",
   "provider": "zhipu",
   "provider_display": "Zhipu AI",
   "score": 0.0,
   "source": "open-llm-leaderboard-v2"
  }
 ],
 "page": {
  "aliases": [
   "Instruction-Following Eval",
   "Instruction-Following Evaluation for Large Language Models"
  ],
  "category": "instruction-following",
  "contamination": {
   "note": "The prompt set and its verification code are fully public on Hugging Face and GitHub, and the benchmark has been in wide use since late 2023, so specific prompts and expected behaviours are likely present in more recent pretraining corpora. Because scoring is mechanical rather than answer-matched, memorising a plausible-sounding response is less obviously rewarded than on a fixed-answer benchmark, which may blunt but does not eliminate this risk.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "license": "CC BY 4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 500,
   "size_note": "Approximately 500 prompts, each built from one or more of 25 types of verifiable instruction.",
   "splits": "single set (the lm-evaluation-harness implementation reads it from the \"train\" split)",
   "url": "https://huggingface.co/datasets/google/IFEval"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 1b, slice M",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "ifeval",
   "opencompass": "",
   "other": "Reference scoring code ships in the paper's own repository (google-research/instruction_following_eval). In lm-evaluation-harness, the `ifeval` task reads the `google/IFEval` dataset's \"train\" split, generates zero-shot with greedy decoding (temperature 0), and reports four metrics: prompt_level_strict_acc, inst_level_strict_acc, prompt_level_loose_acc and inst_level_loose_acc.\n"
  },
  "id": "ifeval",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "IFEval tests whether a model follows explicit, machine-checkable instructions layered onto a prompt, such as \"write more than 400 words,\" \"mention the keyword 'AI' at least 3 times,\" or \"wrap your answer in double quotation marks.\" Each of the roughly 500 prompts combines one or more of 25 instruction types covering things like length, keyword use, formatting, punctuation and casing. Because every instruction can be checked by a program rather than a human or another model, scoring needs no subjective judgment call, unlike most instruction-following or helpfulness evaluations.\n",
  "metric": {
   "baseline_note": "No formal random or human baseline published. Four related scores are reported per model: strict and loose accuracy, each computed both per-prompt (every instruction in the prompt must pass) and per-instruction (each instruction scored independently); \"loose\" scoring allows minor, immaterial response variations (such as markdown wrappers) that \"strict\" scoring does not.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "instruction-following accuracy (strict and loose, at the prompt level and the instruction level)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "IFEval",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2311.07911",
   "title": "Instruction-Following Evaluation for Large Language Models",
   "url": "https://arxiv.org/abs/2311.07911",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Jeffrey Zhou",
    "Tianjian Lu",
    "Swaroop Mishra",
    "Siddhartha Brahma",
    "Sujoy Basu",
    "Yi Luan",
    "Denny Zhou",
    "Le Hou"
   ],
   "org": "Google Research",
   "url": "https://github.com/google-research/google-research/tree/master/instruction_following_eval"
  },
  "released": "2023-11",
  "repo_url": "https://github.com/google-research/google-research/tree/master/instruction_following_eval",
  "saturation": {
   "as_of": "",
   "note": "No current leaderboard was opened for this page, so a present-day top score is not established here. IFEval is widely reported (this repository's own model cards show scores well above 90% for several current models), which is at least suggestive of a benchmark under ceiling pressure, but that was not independently confirmed against a primary leaderboard in this research pass.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Instruction-Following Evaluation for Large Language Models",
    "url": "https://arxiv.org/abs/2311.07911"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness: ifeval task config",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/ifeval/ifeval.yaml"
   }
  ],
  "status": "active",
  "subcategory": "objectively verifiable natural-language instruction compliance",
  "summary": "IFEval scores whether a model's response obeys objectively checkable instructions, such as word counts or keyword frequency, using code rather than human or LLM judgment.",
  "tags": [
   "instruction-following",
   "verifiable",
   "zero-shot",
   "text"
  ],
  "task_format": "A model is given a prompt containing one or more verifiable instructions and generates a free-form response in a single turn, zero-shot, with no few-shot examples. A separate verification function checks the response against each instruction mechanically (for example, counting words or scanning for a keyword) and returns pass/fail per instruction.\n"
 }
}