{
 "body": "\n## What it measures\n\nBIG-Bench Hard (BBH) is a curated subset of 23 tasks pulled out of the much larger BIG-Bench collection, selected because, when BBH was assembled, no language model evaluated on BIG-Bench had beaten the average human rater on them. The 23 tasks are individually varied: logical deduction and object tracking (following several entities through a sequence of swaps or moves), causal judgement, date and multistep arithmetic, geometric shape recognition from SVG-like path descriptions, and language tasks like disambiguating an ambiguous pronoun or spotting sarcasm. What ties them together is not a single skill but the fact that solving any of them reliably requires chaining several reasoning steps rather than pattern-matching a single fact.\n\nBecause the tasks are so heterogeneous, a single averaged BBH score is best read as a broad reasoning-robustness signal rather than a measurement of one specific capability.\n\n## How it is scored\n\nEach of the 23 tasks is scored by exact-match accuracy against its own fixed set of examples (250 examples for most tasks, fewer for three tasks), and the headline BBH number is the accuracy averaged across all 23. The original paper's central finding was that chain-of-thought prompting made a large difference on this suite: it let PaLM exceed average human-rater performance on 10 of the 23 tasks, and let Codex (code-davinci-002) exceed it on 17 of the 23, compared to far fewer under standard direct prompting. Because of this, whether a reported BBH score used chain-of-thought prompting, and how many few-shot examples were given, materially changes the number, and different harnesses (bbh_cot_fewshot vs bbh_zeroshot, for instance) are not directly comparable.\n\n## Dataset and licence\n\nBBH totals 6,511 examples spread across its 23 tasks; most tasks contribute exactly 250 examples, with three exceptions running smaller (penguins_in_a_table at 146, causal_judgement at 187, snarks at 178). There is no train/validation/test split within a task; every example exists only to be evaluated once. The GitHub repository, which holds both the task data and the code-davinci-002 outputs and chain-of-thought prompts used in the paper, is released under the MIT licence.\n\n## Who publishes it\n\nBBH comes from a Google Research team: Mirac Suzgun, Nathan Scales, Nathanael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou and Jason Wei, published as \"Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them\" in October 2022. The dataset and code are maintained in Suzgun's GitHub repository rather than through a dedicated hosted leaderboard.\n\n## Lineage\n\nBBH is a hard subset drawn from the much larger BIG-Bench collection, which is not itself a page in this repository. It does not have its own family of per-task subset pages in this repository; all 23 tasks are represented here only as the single averaged `bbh` score. BBH was later adopted as one of the harder replacement tasks (alongside `mmlu_pro`, `gpqa_diamond`, `musr`, `ifeval` and MATH Level 5) when the Hugging Face Open LLM Leaderboard moved to its v2 iteration in 2024, specifically to replace benchmarks like `arc_challenge`, `hellaswag` and the original MMLU that had become saturated.\n\n## Saturation and contamination\n\nBBH was picked as a v2 replacement benchmark specifically for being harder to saturate than the benchmarks it replaced, and no current ceiling figure was confirmed from a source opened during this research; the Hugging Face leaderboard that most recently tracked it at scale has itself since been archived, which limits how easy it is to check a current, broadly comparable top score. Because every task's examples and answers are public in the GitHub repository and mirrored on Hugging Face, contamination through web-scale pretraining is plausible for any model trained since 2022, though no source read here demonstrated it directly for a specific model.\n\n## How to run it\n\nlm-evaluation-harness exposes several BBH variants: `bbh` (an alias for `bbh_cot_fewshot`, its chain-of-thought few-shot default), plus `bbh_zeroshot`, `bbh_fewshot` and `bbh_cot_zeroshot`. Because the original paper's headline result was specifically about chain-of-thought prompting mattering a great deal on this suite, scores from a zero-shot, non-CoT run and a few-shot CoT run on the same model can differ substantially, and reports rarely state which variant they used unless the harness task name is quoted directly.\n\n## Reading the numbers\n\nA high BBH score suggests a model can sustain multi-step reasoning across a genuinely varied set of small puzzle-like tasks, not just one narrow skill, which makes it a reasonable broad checkpoint. Because the suite averages 23 dissimilar tasks into one number, two models with the same overall BBH score can have very different strengths and weaknesses underneath it, so a large gap on any single sub-task is worth checking directly rather than trusting the average. As with any benchmark whose data has been public for several years, treat an unusually high score with some caution absent corroboration from a fresher or held-out reasoning benchmark.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "bbh",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1",
   "model_id": "deepseek/deepseek-r1",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 68.7,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 0528",
   "model_id": "deepseek/deepseek-r1-0528",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 68.7,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 0528 NVFP4 v2",
   "model_id": "nvidia/deepseek-r1-0528-nvfp4-v2",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 68.7,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek Reasoner",
   "model_id": "deepseek/deepseek-reasoner",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 68.7,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 72B Instruct",
   "model_id": "qwen/qwen2-5-72b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 66.3,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 32B Instruct",
   "model_id": "qwen/qwen2-5-coder-32b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 66.3,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B Instruct",
   "model_id": "meta/meta-llama-3-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 65.5,
   "source": "llm-stats, open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B Instruct",
   "model_id": "nous-research/meta-llama-3-70b-instruct",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 65.5,
   "source": "llm-stats, open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B",
   "model_id": "meta/meta-llama-3-70b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 64.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek Chat",
   "model_id": "deepseek/deepseek-chat",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 64.1,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V2",
   "model_id": "deepseek/deepseek-v2",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 64.1,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V2 Lite",
   "model_id": "deepseek/deepseek-v2-lite",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 64.1,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V2 Lite Chat",
   "model_id": "deepseek/deepseek-v2-lite-chat",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 64.1,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3",
   "model_id": "deepseek/deepseek-v3",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 64.1,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, domain-evals open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3 0324",
   "model_id": "deepseek/deepseek-v3-0324",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 64.1,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3.1",
   "model_id": "deepseek/deepseek-v3-1",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 64.1,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3.2",
   "model_id": "deepseek/deepseek-v3-2",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 64.1,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3.2 Exp",
   "model_id": "deepseek/deepseek-v3-2-exp",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 64.1,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 32B Instruct AWQ",
   "model_id": "qwen/qwen2-5-coder-32b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 64.0,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 32B",
   "model_id": "qwen/qwen3-32b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 62.8,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 32B AWQ",
   "model_id": "qwen/qwen3-32b-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 62.8,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 32B NVFP4",
   "model_id": "nvidia/qwen3-32b-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 62.8,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 31B",
   "model_id": "google/gemma-4-31b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 62.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 31B it",
   "model_id": "google/gemma-4-31b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 62.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 31B it GGUF",
   "model_id": "unsloth/gemma-4-31b-it-gguf",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 62.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 31B IT NVFP4",
   "model_id": "nvidia/gemma-4-31b-it-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 62.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 14B Instruct",
   "model_id": "qwen/qwen2-5-coder-14b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 61.4,
   "source": "bigcode-leaderboard, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B Chat 16K",
   "model_id": "01-ai/yi-1-5-34b-chat-16k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 61.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B 32K",
   "model_id": "01-ai/yi-1-5-34b-32k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 60.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 26B",
   "model_id": "google/gemma-4-26b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 60.1,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large (latest)",
   "model_id": "mistral/mistral-large-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 58.9,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large 2.1",
   "model_id": "mistral/mistral-large-2411",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 58.9,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals,, domain-evals open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large 3",
   "model_id": "mistral/mistral-large-2512",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 58.9,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 32B Instruct",
   "model_id": "qwen/qwen2-5-32b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 58.4,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 32B Instruct AWQ",
   "model_id": "qwen/qwen2-5-32b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 58.4,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.3 70B Instruct NVFP4",
   "model_id": "nvidia/llama-3-3-70b-instruct-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 58.1,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-3.3-70B-Instruct",
   "model_id": "meta/llama-3-3-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 58.1,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Llama 70B",
   "model_id": "deepseek/deepseek-r1-distill-llama-70b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 57.2,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Phi 4 mini instruct",
   "model_id": "microsoft/phi-4-mini-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 56.9,
   "source": "lmarena.ai, provider-reports, preference-evals, llm-stats, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 7B Instruct",
   "model_id": "tii/falcon3-7b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 56.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 27B",
   "model_id": "google/gemma-3-27b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 56.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2,, domain-evals llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 70B",
   "model_id": "meta/llama-3-1-70b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 55.8,
   "source": "lmarena.ai, provider-reports, safety-evals, domain-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 70B Instruct",
   "model_id": "meta/llama-3-1-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 55.8,
   "source": "lmarena.ai, provider-reports, safety-evals, domain-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3 mini 128K instruct",
   "model_id": "microsoft/phi-3-mini-128k-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 55.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B Chat",
   "model_id": "01-ai/yi-1-5-9b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 55.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 34B Chat",
   "model_id": "01-ai/yi-34b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 55.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes 2 Mixtral 8x7B DPO",
   "model_id": "nous-research/nous-hermes-2-mixtral-8x7b-dpo",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 55.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "glm 4 9B",
   "model_id": "zhipu/glm-4-9b",
   "provider": "zhipu",
   "provider_display": "Zhipu AI",
   "score": 55.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "GLM 4 9B 0414",
   "model_id": "zhipu/glm-4-9b-0414",
   "provider": "zhipu",
   "provider_display": "Zhipu AI",
   "score": 55.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Infinity Instruct 3M 0625 Yi 1.5 9B",
   "model_id": "baai/infinity-instruct-3m-0625-yi-1-5-9b",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 55.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 14B",
   "model_id": "qwen/qwen3-14b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 55.1,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 14B AWQ",
   "model_id": "qwen/qwen3-14b-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 55.1,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 14B NVFP4",
   "model_id": "nvidia/qwen3-14b-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 55.1,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3 mini 4K instruct",
   "model_id": "microsoft/phi-3-mini-4k-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 55.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 70B hf",
   "model_id": "meta/llama-2-70b-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 54.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 VL 7B Instruct",
   "model_id": "qwen/qwen2-vl-7b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 54.6,
   "source": "open-llm-leaderboard-v2, multimodal-evals"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 VL 7B Instruct AWQ",
   "model_id": "qwen/qwen2-vl-7b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 54.6,
   "source": "open-llm-leaderboard-v2, multimodal-evals"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 34B",
   "model_id": "01-ai/yi-34b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 54.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 34B 200K",
   "model_id": "01-ai/yi-34b-200k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 54.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "phi 4",
   "model_id": "microsoft/phi-4",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 54.3,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2,, domain-evals llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Phi 4 multimodal instruct",
   "model_id": "microsoft/phi-4-multimodal-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 54.3,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes 2 SOLAR 10.7B",
   "model_id": "nous-research/nous-hermes-2-solar-10-7b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 54.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2 27B it",
   "model_id": "google/gemma-2-27b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 52.4,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 2 Theta Llama 3 8B",
   "model_id": "nous-research/hermes-2-theta-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 52.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "granite 3.0 8B instruct",
   "model_id": "ibm/granite-3-0-8b-instruct",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 51.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 3 Llama 3.1 8B",
   "model_id": "nous-research/hermes-3-llama-3-1-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 51.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 3 Llama 3.1 8B GGUF",
   "model_id": "nous-research/hermes-3-llama-3-1-8b-gguf",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 51.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B Chat 16K",
   "model_id": "01-ai/yi-1-5-9b-chat-16k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 51.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B",
   "model_id": "01-ai/yi-1-5-9b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 51.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 14B Instruct",
   "model_id": "qwen/qwen2-5-14b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 51.2,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 14B Instruct AWQ",
   "model_id": "qwen/qwen2-5-14b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 51.2,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 7B Base",
   "model_id": "tii/falcon3-7b-base",
   "provider": "tii",
   "provider_display": "TII",
   "score": 51.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Infinity Instruct 7M Gen Llama3 1 8B",
   "model_id": "baai/infinity-instruct-7m-gen-llama3-1-8b",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 50.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 2 Pro Llama 3 8B",
   "model_id": "nous-research/hermes-2-pro-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 50.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 30B A3B Instruct 2507",
   "model_id": "qwen/qwen3-30b-a3b-instruct-2507",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 50.5,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 30B A3B NVFP4",
   "model_id": "nvidia/qwen3-30b-a3b-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 50.5,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 30B-A3B",
   "model_id": "qwen/qwen3-30b-a3b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 50.5,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Qwen 32B",
   "model_id": "deepseek/deepseek-r1-distill-qwen-32b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 50.1,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 7B Instruct",
   "model_id": "qwen/qwen2-5-coder-7b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 50.0,
   "source": "bigcode-leaderboard, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B 32K",
   "model_id": "01-ai/yi-1-5-9b-32k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 49.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Infinity Instruct 3M 0625 Llama3 8B",
   "model_id": "baai/infinity-instruct-3m-0625-llama3-8b",
   "provider": "baai",
   "provider_display": "BAAI",
   "score": 49.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OpenHermes 2 Mistral 7B",
   "model_id": "teknium/openhermes-2-mistral-7b",
   "provider": "teknium",
   "provider_display": "Teknium",
   "score": 49.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 9B",
   "model_id": "01-ai/yi-9b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 49.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B Instruct",
   "model_id": "meta/meta-llama-3-8b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 49.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B Instruct",
   "model_id": "nous-research/meta-llama-3-8b-instruct",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 49.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "phi 2",
   "model_id": "microsoft/phi-2",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 48.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 12B",
   "model_id": "google/gemma-3-12b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 48.7,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats, multimodal-evals"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OpenHermes 2.5 Mistral 7B",
   "model_id": "teknium/openhermes-2-5-mistral-7b",
   "provider": "teknium",
   "provider_display": "Teknium",
   "score": 48.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 7B Instruct GPTQ Int4",
   "model_id": "qwen/qwen2-5-coder-7b-instruct-gptq-int4",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 48.6,
   "source": "bigcode-leaderboard, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x22B",
   "model_id": "mistral/open-mixtral-8x22b",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 48.5,
   "source": "open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x22B Instruct v0.1",
   "model_id": "mistral/mixtral-8x22b-instruct-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 48.5,
   "source": "open-llm-leaderboard-v2, llm-stats, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 8B",
   "model_id": "qwen/qwen3-8b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 48.3,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 8B AWQ",
   "model_id": "qwen/qwen3-8b-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 48.3,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 8B Base",
   "model_id": "qwen/qwen3-8b-base",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 48.3,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B",
   "model_id": "01-ai/yi-1-5-34b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 48.2,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B Chat",
   "model_id": "01-ai/yi-1-5-34b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 48.2,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Yi Coder 9B",
   "model_id": "01-ai/yi-coder-9b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 48.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Yi Coder 9B Chat",
   "model_id": "01-ai/yi-coder-9b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 48.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Yi 9B 200K",
   "model_id": "01-ai/yi-9b-200k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 47.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B",
   "model_id": "cerebras/llama3-1-8b",
   "provider": "cerebras",
   "provider_display": "Cerebras",
   "score": 47.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Ministral 8B Instruct 2410",
   "model_id": "mistral/ministral-8b-instruct-2410",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 47.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 3B Instruct",
   "model_id": "tii/falcon3-3b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 47.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "glm 4 9B chat",
   "model_id": "zhipu/glm-4-9b-chat",
   "provider": "zhipu",
   "provider_display": "Zhipu AI",
   "score": 47.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Command R+",
   "model_id": "cohere/command-r-plus-08-2024",
   "provider": "cohere",
   "provider_display": "Cohere",
   "score": 47.3,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B Instruct v0.3",
   "model_id": "mistral/mistral-7b-instruct-v0-3",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 47.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 3B Instruct",
   "model_id": "qwen/qwen2-5-3b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 46.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 Mamba 7B Instruct",
   "model_id": "tii/falcon3-mamba-7b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 46.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 3B Instruct",
   "model_id": "meta/llama-3-2-3b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 46.1,
   "source": "llm-stats, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B",
   "model_id": "meta/meta-llama-3-8b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 46.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B",
   "model_id": "nous-research/meta-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 46.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 6B Chat",
   "model_id": "01-ai/yi-1-5-6b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 45.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "flan t5 xl",
   "model_id": "google/flan-t5-xl",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 45.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B v0.3",
   "model_id": "mistral/mistral-7b-v0-3",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 45.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "mistral 7B v0.3 bnb 4bit",
   "model_id": "unsloth/mistral-7b-v0-3-bnb-4bit",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 45.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 6B",
   "model_id": "01-ai/yi-1-5-6b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 44.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B Instruct v0.2",
   "model_id": "mistral/mistral-7b-instruct-v0-2",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 44.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "glm 4 9B chat hf",
   "model_id": "zhipu/glm-4-9b-chat-hf",
   "provider": "zhipu",
   "provider_display": "Zhipu AI",
   "score": 44.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 3B Base",
   "model_id": "tii/falcon3-3b-base",
   "provider": "tii",
   "provider_display": "TII",
   "score": 44.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B v0.1",
   "model_id": "mistral/mistral-7b-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 44.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Nemo",
   "model_id": "mistral/mistral-nemo",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 44.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Nemo Base 2407",
   "model_id": "mistral/mistral-nemo-base-2407",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 44.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Nemo Instruct 2407",
   "model_id": "mistral/mistral-nemo-instruct-2407",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 44.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "granite 3.1 2B instruct",
   "model_id": "ibm/granite-3-1-2b-instruct",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 44.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2 2B it",
   "model_id": "google/gemma-2-2b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 44.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 11B",
   "model_id": "tii/falcon-11b",
   "provider": "tii",
   "provider_display": "TII",
   "score": 43.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 7B",
   "model_id": "qwen/qwen2-5-7b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 43.8,
   "source": "provider-reports, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 7B Instruct",
   "model_id": "qwen/qwen2-5-7b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 43.8,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 7B Instruct AWQ",
   "model_id": "qwen/qwen2-5-7b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 43.8,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Qwen 14B",
   "model_id": "deepseek/deepseek-r1-distill-qwen-14b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 43.5,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 3 Llama 3.2 3B",
   "model_id": "nous-research/hermes-3-llama-3-2-3b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 43.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B",
   "model_id": "01-ai/yi-6b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 43.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 1.5B Instruct",
   "model_id": "qwen/qwen2-5-1-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 42.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B 200K",
   "model_id": "01-ai/yi-6b-200k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 42.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "falcon mamba 7B",
   "model_id": "tii/falcon-mamba-7b",
   "provider": "tii",
   "provider_display": "TII",
   "score": 42.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "falcon mamba 7B instruct",
   "model_id": "tii/falcon-mamba-7b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 42.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "falcon mamba 7B instruct Q4 K M GGUF",
   "model_id": "tii/falcon-mamba-7b-instruct-q4-k-m-gguf",
   "provider": "tii",
   "provider_display": "TII",
   "score": 42.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2 9B",
   "model_id": "google/gemma-2-9b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 42.8,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2 9B it",
   "model_id": "google/gemma-2-9b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 42.8,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OpenHermes 13B",
   "model_id": "teknium/openhermes-13b",
   "provider": "teknium",
   "provider_display": "Teknium",
   "score": 42.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "glm 4 9B chat 1M",
   "model_id": "zhipu/glm-4-9b-chat-1m",
   "provider": "zhipu",
   "provider_display": "Zhipu AI",
   "score": 41.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "flan t5 large",
   "model_id": "google/flan-t5-large",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 41.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 13B hf",
   "model_id": "meta/llama-2-13b-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 41.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 13B hf",
   "model_id": "nous-research/llama-2-13b-hf",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 41.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B Chat",
   "model_id": "01-ai/yi-6b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 41.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B Chat 4bits",
   "model_id": "01-ai/yi-6b-chat-4bits",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 41.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 1.5B",
   "model_id": "qwen/qwen2-5-1-5b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 40.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 1.5B Instruct AWQ",
   "model_id": "qwen/qwen2-5-1-5b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 40.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 40B instruct",
   "model_id": "tii/falcon-40b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 40.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 40B",
   "model_id": "tii/falcon-40b",
   "provider": "tii",
   "provider_display": "TII",
   "score": 40.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OLMo 2 1124 7B",
   "model_id": "allen-ai/olmo-2-1124-7b",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 40.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Llama 8B",
   "model_id": "deepseek/deepseek-r1-distill-llama-8b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 39.8,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "granite 3.1 8B instruct",
   "model_id": "ibm/granite-3-1-8b-instruct",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 39.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B",
   "model_id": "meta/llama-3-1-8b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 39.2,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct",
   "model_id": "meta/llama-3-1-8b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 39.2,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct",
   "model_id": "unsloth/llama-3-1-8b-instruct",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 39.2,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct FP8",
   "model_id": "nvidia/llama-3-1-8b-instruct-fp8",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 39.2,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct NVFP4",
   "model_id": "nvidia/llama-3-1-8b-instruct-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 39.2,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 3B",
   "model_id": "meta/llama-3-2-3b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 39.1,
   "source": "llm-stats, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "stablelm zephyr 3B",
   "model_id": "stability/stablelm-zephyr-3b",
   "provider": "stability",
   "provider_display": "Stability AI",
   "score": 38.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Command R",
   "model_id": "cohere/command-r-08-2024",
   "provider": "cohere",
   "provider_display": "Cohere",
   "score": 38.5,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 1.5B Instruct",
   "model_id": "qwen/qwen2-1-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 38.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Qwen 7B",
   "model_id": "deepseek/deepseek-r1-distill-qwen-7b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 38.2,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes llama 2 7B",
   "model_id": "nous-research/nous-hermes-llama-2-7b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 38.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OLMoE 1B 7B 0125",
   "model_id": "allen-ai/olmoe-1b-7b-0125",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 38.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OLMoE 1B 7B 0125 Instruct",
   "model_id": "allen-ai/olmoe-1b-7b-0125-instruct",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 38.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3.5 mini instruct",
   "model_id": "microsoft/phi-3-5-mini-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 38.1,
   "source": "open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Math 1.5B",
   "model_id": "qwen/qwen2-5-math-1-5b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 37.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 1B Instruct",
   "model_id": "tii/falcon3-1b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 37.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2 2B",
   "model_id": "google/gemma-2-2b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 37.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x7B",
   "model_id": "mistral/open-mixtral-8x7b",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 36.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x7B Instruct v0.1",
   "model_id": "mistral/mixtral-8x7b-instruct-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 36.8,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x7B v0.1",
   "model_id": "mistral/mixtral-8x7b-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 36.8,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 10B Base",
   "model_id": "tii/falcon3-10b-base",
   "provider": "tii",
   "provider_display": "TII",
   "score": 36.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 10B Instruct",
   "model_id": "tii/falcon3-10b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 36.5,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 7B it",
   "model_id": "google/gemma-7b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 36.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek llm 7B chat",
   "model_id": "deepseek/deepseek-llm-7b-chat",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 36.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Jamba v0.1",
   "model_id": "ai21/jamba-v0-1",
   "provider": "ai21",
   "provider_display": "AI21 Labs",
   "score": 36.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Falcon3 1B Base",
   "model_id": "tii/falcon3-1b-base",
   "provider": "tii",
   "provider_display": "TII",
   "score": 35.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "flan t5 base",
   "model_id": "google/flan-t5-base",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 35.3,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek llm 7B base",
   "model_id": "deepseek/deepseek-llm-7b-base",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 35.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B hf",
   "model_id": "meta/llama-2-7b-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 35.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B hf",
   "model_id": "nous-research/llama-2-7b-hf",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 35.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "stablelm 3B 4e1t",
   "model_id": "stability/stablelm-3b-4e1t",
   "provider": "stability",
   "provider_display": "Stability AI",
   "score": 35.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 1B Instruct",
   "model_id": "meta/llama-3-2-1b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 34.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B 32K Instruct",
   "model_id": "together/llama-2-7b-32k-instruct",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 34.4,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "deepseek moe 16B base",
   "model_id": "deepseek/deepseek-moe-16b-base",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 34.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "LLaMA 2 7B 32K",
   "model_id": "together/llama-2-7b-32k",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 34.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "OLMoE 1B 7B 0924",
   "model_id": "allen-ai/olmoe-1b-7b-0924",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 33.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "RedPajama INCITE 7B Instruct",
   "model_id": "together/redpajama-incite-7b-instruct",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 33.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "stablelm 2 1 6B",
   "model_id": "stability/stablelm-2-1-6b",
   "provider": "stability",
   "provider_display": "Stability AI",
   "score": 33.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2B",
   "model_id": "google/gemma-2b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 33.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B Instruct v0.1",
   "model_id": "mistral/mistral-7b-instruct-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 33.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 13B chat hf",
   "model_id": "meta/llama-2-13b-chat-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 33.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "GPT NeoXT Chat Base 20B",
   "model_id": "together/gpt-neoxt-chat-base-20b",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 33.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 0.5B Instruct",
   "model_id": "qwen/qwen2-5-0-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 33.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "rwkv raven 14B",
   "model_id": "rwkv/rwkv-raven-14b",
   "provider": "rwkv",
   "provider_display": "RWKV Foundation",
   "score": 33.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "GPT JT 6B v1",
   "model_id": "together/gpt-jt-6b-v1",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 33.0,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 7B",
   "model_id": "tii/falcon-7b",
   "provider": "tii",
   "provider_display": "TII",
   "score": 32.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "flan t5 small",
   "model_id": "google/flan-t5-small",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 32.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "granite 3.1 1B a400m instruct",
   "model_id": "ibm/granite-3-1-1b-a400m-instruct",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 32.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 0.5B",
   "model_id": "qwen/qwen2-5-0-5b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 32.7,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Qwen 1.5B",
   "model_id": "deepseek/deepseek-r1-distill-qwen-1-5b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 32.4,
   "source": "bigcode-leaderboard, provider-reports, open-llm-leaderboard, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "granite 3.0 1B a400m base",
   "model_id": "ibm/granite-3-0-1b-a400m-base",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 32.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "RedPajama INCITE Chat 3B v1",
   "model_id": "together/redpajama-incite-chat-3b-v1",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 32.2,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 7B instruct",
   "model_id": "tii/falcon-7b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 32.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Ministral 3B (latest)",
   "model_id": "mistral/ministral-3b-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 31.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "RedPajama INCITE 7B Base",
   "model_id": "together/redpajama-incite-7b-base",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 31.9,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "gemma 1.1 2B it",
   "model_id": "google/gemma-1-1-2b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 31.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "RedPajama INCITE 7B Chat",
   "model_id": "together/redpajama-incite-7b-chat",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 31.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 0.5B Instruct",
   "model_id": "qwen/qwen2-0-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 31.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2B it",
   "model_id": "google/gemma-2b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 31.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B chat hf",
   "model_id": "meta/llama-2-7b-chat-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 31.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B chat hf",
   "model_id": "nous-research/llama-2-7b-chat-hf",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 31.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 1B",
   "model_id": "meta/llama-3-2-1b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 31.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 1B",
   "model_id": "nous-research/llama-3-2-1b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 31.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "RedPajama INCITE Base 3B v1",
   "model_id": "together/redpajama-incite-base-3b-v1",
   "provider": "together",
   "provider_display": "Together AI",
   "score": 30.6,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "OLMo 1B hf",
   "model_id": "allen-ai/olmo-1b-hf",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 30.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 70B chat hf",
   "model_id": "meta/llama-2-70b-chat-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 30.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "DialoGPT medium",
   "model_id": "microsoft/dialogpt-medium",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 30.1,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "mt5 base",
   "model_id": "google/mt5-base",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 28.8,
   "source": "open-llm-leaderboard-v2"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "mt5 small",
   "model_id": "google/mt5-small",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 27.7,
   "source": "open-llm-leaderboard-v2"
  }
 ],
 "page": {
  "aliases": [
   "BBH"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "All 23 task files and answers are public in the GitHub repository and widely mirrored on Hugging Face, so exposure through web-scale pretraining is plausible; no source read during this research demonstrated specific memorization.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 6511,
   "size_note": "6,511 examples across 23 tasks; most tasks have 250 examples each, with three smaller exceptions (penguins_in_a_table 146, causal_judgement 187, snarks 178)",
   "splits": "no train/test split; each task is a single fixed evaluation set",
   "url": "https://github.com/suzgunmirac/BIG-Bench-Hard"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice F",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "bbh",
   "opencompass": "",
   "other": "lm-evaluation-harness also exposes bbh_zeroshot, bbh_fewshot, bbh_cot_fewshot and bbh_cot_zeroshot variants; the bare bbh group is an alias for bbh_cot_fewshot"
  },
  "id": "bbh",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "BIG-Bench Hard bundles 23 tasks selected from the much larger BIG-Bench collection, chosen because, at the time of selection, no language model in the original BIG-Bench evaluation had outperformed the average human rater on them. The tasks span logical deduction, causal judgement, object tracking and counting, date and arithmetic reasoning, and several language-understanding puzzles such as disambiguation and sarcasm detection (snarks), so the suite is best read as a stress test of multi-step reasoning under diverse, individually simple-looking task formats rather than a single coherent skill.",
  "metric": {
   "baseline_note": "Tasks were selected specifically because average human-rater performance exceeded the best contemporary language model at the time BBH was created; no single averaged human-baseline number across all 23 tasks was confirmed from a source opened during this research.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy (exact match), averaged across the 23 tasks",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "BIG-Bench Hard",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2210.09261",
   "title": "Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them",
   "url": "https://arxiv.org/abs/2210.09261",
   "year": 2022
  },
  "publisher": {
   "authors": [
    "Mirac Suzgun",
    "Nathan Scales",
    "Nathanael Scharli",
    "Sebastian Gehrmann",
    "Yi Tay",
    "Hyung Won Chung",
    "Aakanksha Chowdhery",
    "Quoc V. Le",
    "Ed H. Chi",
    "Denny Zhou",
    "Jason Wei"
   ],
   "org": "Google Research",
   "url": "https://github.com/suzgunmirac/BIG-Bench-Hard"
  },
  "released": "2022-10",
  "repo_url": "https://github.com/suzgunmirac/BIG-Bench-Hard",
  "saturation": {
   "as_of": "",
   "note": "BBH was adopted as one of the harder replacement benchmarks in the Hugging Face Open LLM Leaderboard v2 (2024) specifically for its resistance to the saturation that had hit ARC-Challenge, HellaSwag and MMLU; no current top-score figure was confirmed from a source opened during this research, and that leaderboard has itself since been archived.",
   "status": "watch",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them",
    "url": "https://arxiv.org/abs/2210.09261"
   },
   {
    "accessed": "2026-09-07",
    "title": "GitHub - suzgunmirac/BIG-Bench-Hard",
    "url": "https://github.com/suzgunmirac/BIG-Bench-Hard"
   },
   {
    "accessed": "2026-09-07",
    "title": "BIG-Bench-Hard README",
    "url": "https://github.com/suzgunmirac/BIG-Bench-Hard/blob/main/README.md"
   },
   {
    "accessed": "2026-09-07",
    "title": "maveriq/bigbenchhard dataset card",
    "url": "https://huggingface.co/datasets/maveriq/bigbenchhard"
   },
   {
    "accessed": "2026-09-07",
    "title": "lm-evaluation-harness BBH task README",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/bbh/README.md"
   },
   {
    "accessed": "2026-09-07",
    "title": "Open LLM Leaderboard (archived) space",
    "url": "https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard"
   }
  ],
  "status": "active",
  "subcategory": "multi-step reasoning suite",
  "summary": "A 23-task suite pulled from BIG-Bench specifically because prior language models failed to beat average human raters on them.",
  "tags": [
   "reasoning",
   "chain-of-thought",
   "multi-task",
   "legacy-benchmark"
  ],
  "task_format": "A mix of multiple-choice and short free-response prompts, varying by task; most are answered directly or via chain-of-thought prompting before a final answer."
 }
}