{
 "body": "\n## What it measures\n\nGSM8K tests whether a language model can solve grade-school-level math word problems that need several linked arithmetic steps rather than one calculation. Each item gives a short English story problem -- for example, working out how much change is left after a series of purchases -- and the model must produce a chain of reasoning that ends in a single numeric answer. OpenAI built the set so that \"a bright middle school student\" could solve every problem, deliberately staying below algebra or calculus. The skill under test is multi-step arithmetic planning and consistent calculation, not advanced mathematics.\n\n## How it is scored\n\nModels are scored by exact match on the final numeric answer, which reference solutions mark with a `####` delimiter after a written chain of reasoning; there is no partial credit for correct intermediate steps. The original paper evaluated fine-tuned GPT-3-scale models, with and without a learned verifier that reranked multiple sampled solutions. Later work standardized on zero-shot or few-shot chain-of-thought prompting with greedy decoding, which is how most model cards now report it. Because the paper does not fix a single prompt format, shot count and chain-of-thought phrasing vary between reporters, and self-consistency decoding (sampling many chains and voting) can add several points over greedy decoding.\n\n## Dataset and licence\n\nThe public release holds 7,473 training problems and 1,319 test problems, each paired with a natural-language solution that carries calculator-style annotations (`<<...>>`) for intermediate arithmetic. A parallel \"Socratic\" version of each split adds auto-generated guiding sub-questions. Problems were written by human problem writers and checked through multiple rounds of worker-agreement review. OpenAI publishes the data and code on GitHub and Hugging Face under the MIT licence, and the test-set answers are public rather than held out.\n\n## Who publishes it\n\nGSM8K comes from Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse and John Schulman at OpenAI, published as \"Training Verifiers to Solve Math Word Problems\" on arXiv in October 2021. OpenAI maintains the reference dataset and scoring code at github.com/openai/grade-school-math. There is no OpenAI-run public leaderboard, but third parties score it continuously and most model cards report a GSM8K number.\n\n## Lineage\n\nGSM8K does not name a predecessor benchmark of its own. Reaching its difficulty ceiling prompted harder or contamination-resistant follow-ups outside this repository's current pages, including Scale AI's GSM1k -- a matched-difficulty, held-out replacement built specifically to test overfitting on GSM8K -- and Apple's GSM-Symbolic, which perturbs GSM8K-style problems to probe brittleness. Neither has a page in this repository yet, and no successor or variant id is tracked here.\n\n## Saturation and contamination\n\nMultiple frontier models now cluster at or above 99% accuracy: a third-party leaderboard aggregator put Baidu's ERNIE 5.0 at 99.7% as of April 2026, with several other frontier models at or near 99% the same month, leaving little room to separate current top systems. A 2024 NeurIPS study (the GSM1k paper) tested this directly by writing a fresh, matched-difficulty test set: accuracy dropped by up to 8 points on it, and the size of the drop correlated with how likely a model was to regenerate GSM8K items verbatim, evidence of memorization in several model families. The same study found frontier models showed comparatively little of this effect, so the ceiling looks more like a genuine capability limit for top models than pure memorization -- though the four-plus-year-old, publicly answered test set remains a plausible contamination vector for any model trained on general web text.\n\n## How to run it\n\nlm-evaluation-harness implements it as the `gsm8k` task family, including `gsm8k_cot` (chain-of-thought), `gsm8k_cot_self_consistency` (multi-sample voting) and a Llama-specific chat-template variant. OpenAI's own grade-school-math repository ships a reference scorer that strips the calculator annotations and checks the answer after `####`. Because shot count, chain-of-thought instructions and self-consistency sampling all move the score, two reported GSM8K numbers are only comparable when the evaluation protocol matches.\n\n## Reading the numbers\n\nA high GSM8K score by itself now says little about a frontier model, since most of them already sit at or near the ceiling; the benchmark mostly still distinguishes small or older models from current ones. A low score remains meaningful, flagging weak elementary arithmetic or an inconsistent reasoning chain. Given the age of the public test set and the documented memorization gap on GSM1k, treat any single GSM8K number as a rough signal and check harder, newer math benchmarks before drawing conclusions about a model's actual mathematical reasoning.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "gsm8k",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Kimi K2 0711",
   "model_id": "moonshot/kimi-k2-0711-preview",
   "provider": "moonshot",
   "provider_display": "Moonshot AI",
   "score": 97.3,
   "source": "llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Kimi K2 0905",
   "model_id": "moonshot/kimi-k2-0905-preview",
   "provider": "moonshot",
   "provider_display": "Moonshot AI",
   "score": 97.3,
   "source": "llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Kimi K2 Thinking",
   "model_id": "moonshot/kimi-k2-thinking",
   "provider": "moonshot",
   "provider_display": "Moonshot AI",
   "score": 97.3,
   "source": "llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Kimi K2 Thinking Turbo",
   "model_id": "moonshot/kimi-k2-thinking-turbo",
   "provider": "moonshot",
   "provider_display": "Moonshot AI",
   "score": 97.3,
   "source": "llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Kimi K2 Turbo",
   "model_id": "moonshot/kimi-k2-turbo-preview",
   "provider": "moonshot",
   "provider_display": "Moonshot AI",
   "score": 97.3,
   "source": "llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Kimi K2.5",
   "model_id": "moonshot/kimi-k2-5",
   "provider": "moonshot",
   "provider_display": "Moonshot AI",
   "score": 97.3,
   "source": "llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Kimi K2.5 NVFP4",
   "model_id": "nvidia/kimi-k2-5-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 97.3,
   "source": "llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o1",
   "model_id": "openai/o1",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 97.1,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o1-mini",
   "model_id": "openai/o1-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 97.1,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o1-pro",
   "model_id": "openai/o1-pro",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 97.1,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 405B",
   "model_id": "meta/llama-3-1-405b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 96.8,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 405B Instruct",
   "model_id": "meta/llama-3-1-405b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 96.8,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 405B Instruct FP8",
   "model_id": "meta/llama-3-1-405b-instruct-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 96.8,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5",
   "model_id": "anthropic/claude-3-5-sonnet-20240620",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 96.4,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5 v2",
   "model_id": "anthropic/claude-3-5-sonnet-20241022",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 96.4,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 27B",
   "model_id": "google/gemma-3-27b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 95.9,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2,, domain-evals llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 3",
   "model_id": "anthropic/claude-3-opus-20240229",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 95.0,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o",
   "model_id": "openai/gpt-4o",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-05-13)",
   "model_id": "openai/gpt-4o-2024-05-13",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-08-06)",
   "model_id": "openai/gpt-4o-2024-08-06",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-11-20)",
   "model_id": "openai/gpt-4o-2024-11-20",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.3 70B Instruct NVFP4",
   "model_id": "nvidia/llama-3-3-70b-instruct-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 95.0,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-3.3-70B-Instruct",
   "model_id": "meta/llama-3-3-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 95.0,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B Instruct",
   "model_id": "meta/meta-llama-3-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 95.0,
   "source": "llm-stats, open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B Instruct",
   "model_id": "nous-research/meta-llama-3-70b-instruct",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 95.0,
   "source": "llm-stats, open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 12B",
   "model_id": "google/gemma-3-12b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 94.4,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 235B-A22B",
   "model_id": "qwen/qwen3-235b-a22b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 94.4,
   "source": "lmarena.ai, provider-reports, preference-evals, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "phi 4",
   "model_id": "microsoft/phi-4",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 93.5,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2,, domain-evals llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4 Turbo",
   "model_id": "openai/gpt-4-turbo",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 93.0,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large (latest)",
   "model_id": "mistral/mistral-large-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 93.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3",
   "model_id": "anthropic/claude-3-sonnet-20240229",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 92.3,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4",
   "model_id": "openai/gpt-4",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 92.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o mini",
   "model_id": "openai/gpt-4o-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 92.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 7B",
   "model_id": "qwen/qwen2-5-7b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 91.6,
   "source": "provider-reports, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Pro",
   "model_id": "google/gemini-1-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 90.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro",
   "model_id": "google/gemini-2-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 90.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek Chat",
   "model_id": "deepseek/deepseek-chat",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 89.3,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek Reasoner",
   "model_id": "deepseek/deepseek-reasoner",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 89.3,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3",
   "model_id": "deepseek/deepseek-v3",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 89.3,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, domain-evals open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3 0324",
   "model_id": "deepseek/deepseek-v3-0324",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 89.3,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3.1",
   "model_id": "deepseek/deepseek-v3-1",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 89.3,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3.2",
   "model_id": "deepseek/deepseek-v3-2",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 89.3,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3.2 Exp",
   "model_id": "deepseek/deepseek-v3-2-exp",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 89.3,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 4B",
   "model_id": "google/gemma-3-4b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 89.2,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 3 4B pt",
   "model_id": "google/gemma-3-4b-pt",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 89.2,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3",
   "model_id": "anthropic/claude-3-haiku-20240307",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 88.9,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Phi 4 mini instruct",
   "model_id": "microsoft/phi-4-mini-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 88.6,
   "source": "lmarena.ai, provider-reports, preference-evals, llm-stats, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x22B",
   "model_id": "mistral/open-mixtral-8x22b",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 88.0,
   "source": "open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x22B Instruct v0.1",
   "model_id": "mistral/mixtral-8x22b-instruct-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 88.0,
   "source": "open-llm-leaderboard-v2, llm-stats, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Flash",
   "model_id": "google/gemini-1-5-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 86.2,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Flash-8B",
   "model_id": "google/gemini-1-5-flash-8b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 86.2,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.0 Flash",
   "model_id": "google/gemini-2-0-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 86.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash",
   "model_id": "google/gemini-2-5-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 86.2,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3.5 mini instruct",
   "model_id": "microsoft/phi-3-5-mini-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 86.2,
   "source": "open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B",
   "model_id": "meta/meta-llama-3-70b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 85.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "granite 3.3 8B instruct",
   "model_id": "ibm/granite-3-3-8b-instruct",
   "provider": "ibm",
   "provider_display": "IBM",
   "score": 80.9,
   "source": "llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-3.5-turbo",
   "model_id": "openai/gpt-3-5-turbo",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 80.0,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 3B",
   "model_id": "meta/llama-3-2-3b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 77.7,
   "source": "llm-stats, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 3B Instruct",
   "model_id": "meta/llama-3-2-3b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 77.7,
   "source": "llm-stats, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3 mini 4K instruct",
   "model_id": "microsoft/phi-3-mini-4k-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 74.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2 27B it",
   "model_id": "google/gemma-2-27b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 74.0,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B",
   "model_id": "01-ai/yi-1-5-34b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 73.2,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B Chat",
   "model_id": "01-ai/yi-1-5-9b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 71.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes 2 Mixtral 8x7B DPO",
   "model_id": "nous-research/nous-hermes-2-mixtral-8x7b-dpo",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 71.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B Chat",
   "model_id": "01-ai/yi-1-5-34b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 71.6,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B Chat 16K",
   "model_id": "01-ai/yi-1-5-34b-chat-16k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 71.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Command R+",
   "model_id": "cohere/command-r-plus-08-2024",
   "provider": "cohere",
   "provider_display": "Cohere",
   "score": 70.7,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 2 Theta Llama 3 8B",
   "model_id": "nous-research/hermes-2-theta-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 70.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes 2 Yi 34B",
   "model_id": "nous-research/nous-hermes-2-yi-34b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 70.1,
   "source": "open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3 mini 128K instruct",
   "model_id": "microsoft/phi-3-mini-128k-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 69.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes 2 SOLAR 10.7B",
   "model_id": "nous-research/nous-hermes-2-solar-10-7b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 69.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B Instruct",
   "model_id": "meta/meta-llama-3-8b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 68.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B Instruct",
   "model_id": "nous-research/meta-llama-3-8b-instruct",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 68.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2 9B",
   "model_id": "google/gemma-2-9b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 68.6,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2 9B it",
   "model_id": "google/gemma-2-9b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 68.6,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 2 Pro Llama 3 8B",
   "model_id": "nous-research/hermes-2-pro-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 67.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 6B Chat",
   "model_id": "01-ai/yi-1-5-6b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 67.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 3 1B it",
   "model_id": "google/gemma-3-1b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 62.8,
   "source": "llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 3 1B pt",
   "model_id": "google/gemma-3-1b-pt",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 62.8,
   "source": "llm-stats"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B",
   "model_id": "01-ai/yi-1-5-9b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 62.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x7B Instruct v0.1",
   "model_id": "mistral/mixtral-8x7b-instruct-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 61.1,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B Chat 16K",
   "model_id": "01-ai/yi-1-5-9b-chat-16k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 59.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x7B v0.1",
   "model_id": "mistral/mixtral-8x7b-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 57.6,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 1.5B Instruct",
   "model_id": "qwen/qwen2-1-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 55.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "phi 2",
   "model_id": "microsoft/phi-2",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 55.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 70B hf",
   "model_id": "meta/llama-2-70b-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 54.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 7B it",
   "model_id": "google/gemma-7b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 52.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 34B",
   "model_id": "01-ai/yi-34b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 50.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 6B",
   "model_id": "01-ai/yi-1-5-6b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 49.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 9B",
   "model_id": "01-ai/yi-9b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 49.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek llm 7B base",
   "model_id": "deepseek/deepseek-llm-7b-base",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 45.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek llm 7B chat",
   "model_id": "deepseek/deepseek-llm-7b-chat",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 45.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B",
   "model_id": "meta/meta-llama-3-8b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 45.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B",
   "model_id": "nous-research/meta-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 45.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B Instruct v0.2",
   "model_id": "mistral/mistral-7b-instruct-v0-2",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 40.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B v0.1",
   "model_id": "mistral/mistral-7b-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 37.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 0.5B Instruct",
   "model_id": "qwen/qwen2-0-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 35.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 34B 200K",
   "model_id": "01-ai/yi-34b-200k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 34.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B v0.3",
   "model_id": "mistral/mistral-7b-v0-3",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 34.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "mistral 7B v0.3 bnb 4bit",
   "model_id": "unsloth/mistral-7b-v0-3-bnb-4bit",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 34.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 40B instruct",
   "model_id": "tii/falcon-40b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 34.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 34B Chat",
   "model_id": "01-ai/yi-34b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 31.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B 200K",
   "model_id": "01-ai/yi-6b-200k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 30.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek coder 6.7B instruct",
   "model_id": "deepseek/deepseek-coder-6-7b-instruct",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 26.8,
   "source": "open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 70B chat hf",
   "model_id": "meta/llama-2-70b-chat-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 26.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 13B hf",
   "model_id": "meta/llama-2-13b-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 22.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 13B hf",
   "model_id": "nous-research/llama-2-13b-hf",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 22.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "CodeLlama 34B Instruct hf",
   "model_id": "meta/codellama-34b-instruct-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 21.6,
   "source": "bigcode-leaderboard, provider-reports, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek coder 6.7B base",
   "model_id": "deepseek/deepseek-coder-6-7b-base",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 18.0,
   "source": "open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2B",
   "model_id": "google/gemma-2b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 16.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 13B chat hf",
   "model_id": "meta/llama-2-13b-chat-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 15.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B hf",
   "model_id": "meta/llama-2-7b-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 14.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B hf",
   "model_id": "nous-research/llama-2-7b-hf",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 14.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B Instruct v0.1",
   "model_id": "mistral/mistral-7b-instruct-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 14.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B",
   "model_id": "01-ai/yi-6b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 12.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B Chat",
   "model_id": "01-ai/yi-6b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 12.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B chat hf",
   "model_id": "meta/llama-2-7b-chat-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 7.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Llama 2 7B chat hf",
   "model_id": "nous-research/llama-2-7b-chat-hf",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 7.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes llama 2 7B",
   "model_id": "nous-research/nous-hermes-llama-2-7b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 5.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "CodeLlama 7B hf",
   "model_id": "meta/codellama-7b-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 5.5,
   "source": "open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "CodeLlama 7B Instruct hf",
   "model_id": "meta/codellama-7b-instruct-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 5.5,
   "source": "open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2B it",
   "model_id": "google/gemma-2b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 5.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 7B instruct",
   "model_id": "tii/falcon-7b-instruct",
   "provider": "tii",
   "provider_display": "TII",
   "score": 4.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 7B",
   "model_id": "tii/falcon-7b",
   "provider": "tii",
   "provider_display": "TII",
   "score": 4.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Baichuan 7B",
   "model_id": "baichuan/baichuan-7b",
   "provider": "baichuan",
   "provider_display": "Baichuan",
   "score": 2.8,
   "source": "open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "OLMo 1B hf",
   "model_id": "allen-ai/olmo-1b-hf",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 1.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek coder 1.3B base",
   "model_id": "deepseek/deepseek-coder-1-3b-base",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 1.1,
   "source": "open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek coder 1.3B instruct",
   "model_id": "deepseek/deepseek-coder-1-3b-instruct",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 1.1,
   "source": "open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B 32K",
   "model_id": "01-ai/yi-1-5-34b-32k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 0.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B 32K",
   "model_id": "01-ai/yi-1-5-9b-32k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 0.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  }
 ],
 "page": {
  "aliases": [
   "Grade School Math 8K"
  ],
  "category": "math",
  "contamination": {
   "note": "Test-set answers have been public since 2021. A 2024 NeurIPS study (the GSM1k paper) found accuracy drops of up to 8 points on a freshly written, matched-difficulty set, correlated with how likely a model was to regenerate GSM8K items verbatim -- evidence of memorization in several model families -- though the same study found frontier models showed comparatively little of this effect.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 8792,
   "size_note": "7,473 training problems + 1,319 test problems; a parallel Socratic-format version of each split adds auto-generated guiding sub-questions",
   "splits": "train (7,473) / test (1,319)",
   "url": "https://huggingface.co/datasets/openai/gsm8k"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice F",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "gsm8k",
   "opencompass": "",
   "other": ""
  },
  "id": "gsm8k",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "GSM8K tests whether a model can solve short, English-language math word problems that take two to eight linked arithmetic steps to reach an answer, such as working out a total after several purchases and discounts. OpenAI wrote the problems so that a bright middle-school student could solve every one of them by hand, deliberately excluding algebra, calculus or other advanced mathematics. The skill under test is multi-step arithmetic planning and consistent calculation rather than mathematical sophistication.",
  "metric": {
   "baseline_note": "",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy (exact-match final answer)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "GSM8K",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2110.14168",
   "title": "Training Verifiers to Solve Math Word Problems",
   "url": "https://arxiv.org/abs/2110.14168",
   "year": 2021
  },
  "publisher": {
   "authors": [
    "Karl Cobbe",
    "Vineet Kosaraju",
    "Mohammad Bavarian",
    "Mark Chen",
    "Heewoo Jun",
    "Lukasz Kaiser",
    "Matthias Plappert",
    "Jerry Tworek",
    "Jacob Hilton",
    "Reiichiro Nakano",
    "Christopher Hesse",
    "John Schulman"
   ],
   "org": "OpenAI",
   "url": "https://github.com/openai/grade-school-math"
  },
  "released": "2021-10",
  "repo_url": "https://github.com/openai/grade-school-math",
  "saturation": {
   "as_of": "2026-04",
   "note": "A third-party leaderboard aggregator put several frontier models at or above 99% as of April 2026 (Baidu's ERNIE 5.0 highest at 99.7%), leaving little room to separate current top models.",
   "status": "saturated",
   "top_score": 99.7
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "Training Verifiers to Solve Math Word Problems",
    "url": "https://arxiv.org/abs/2110.14168"
   },
   {
    "accessed": "2026-09-07",
    "title": "openai/gsm8k dataset card",
    "url": "https://huggingface.co/datasets/openai/gsm8k"
   },
   {
    "accessed": "2026-09-07",
    "title": "openai/grade-school-math repository",
    "url": "https://github.com/openai/grade-school-math"
   },
   {
    "accessed": "2026-09-07",
    "title": "grade-school-math LICENSE (MIT)",
    "url": "https://github.com/openai/grade-school-math/blob/master/LICENSE"
   },
   {
    "accessed": "2026-09-07",
    "title": "lm-evaluation-harness gsm8k task README",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/gsm8k/README.md"
   },
   {
    "accessed": "2026-09-07",
    "title": "A Careful Examination of Large Language Model Performance on Grade School Arithmetic (GSM1k)",
    "url": "https://arxiv.org/abs/2405.00332"
   },
   {
    "accessed": "2026-09-07",
    "title": "GSM8K & MATH Benchmark Leaderboard 2025-2026",
    "url": "https://www.codesota.com/llm/gsm8k-math"
   }
  ],
  "status": "active",
  "subcategory": "grade-school arithmetic word problems",
  "summary": "8.5K grade-school math word problems scored by exact match on the final numeric answer, testing multi-step arithmetic reasoning.",
  "tags": [
   "math",
   "word-problems",
   "chain-of-thought",
   "arithmetic",
   "exact-match"
  ],
  "task_format": "Free-response word problem in English; the model produces a reasoning chain ending in one final numeric answer, extracted after a `####` marker and graded by exact match."
 }
}