{
 "body": "\n## What it measures\n\nMGSM gives a model the same 250 grade-school arithmetic word problems used in GSM8K, each\nprofessionally translated by human annotators into ten typologically diverse languages: Spanish,\nFrench, German, Russian, Chinese, Japanese, Thai, Swahili, Bengali and Telugu. The English originals\nship alongside the translations, so the repository holds eleven 250-row files in total. A model\nreads one problem in one language and must produce a final numeric answer, usually via a worked\nchain-of-thought solution. The paper frames this explicitly as a test of multilingual chain-of-thought\nreasoning, not of translation quality: the translation itself is fixed and handed to the model, and\nwhat is scored is whether multi-step arithmetic reasoning still works once the problem text is no\nlonger in English.\n\n## How it is scored\n\nThe lm-evaluation-harness implementation exposes two protocols. `mgsm_direct` asks the model to\nanswer without showing its work. `mgsm_cot_native` gives few-shot exemplars with worked solutions in\nthe same language as the question and asks the model to reason before answering. The source paper\nmainly used six-shot native-language exemplars (fewer for some languages where GPT-3's context limit\nforced truncation) and scored the fraction of problems where the model's final answer, written as an\nArabic numeral, exactly matches the reference. Scores are typically reported per language and then\naveraged across the ten (or eleven, including English) languages. Because direct-vs-chain-of-thought\nand native-vs-English exemplar prompting change scores substantially, two reported MGSM numbers are\nonly comparable once both state which protocol produced them. No human baseline is reported in the\nsource paper.\n\n## Dataset and licence\n\nMGSM does not introduce new problems: all 250 are GSM8K word problems (Cobbe et al., 2021),\ntranslated by human annotators rather than machine translation. There is no train/test split in the\nusual sense; all 250 problems per language are used for evaluation, and a separate small set of\nfew-shot exemplars, also manually translated per language, ships in `exemplars.py`. The data is\nplain-text TSV (problem, answer) and is licensed CC-BY-4.0, per the LICENSE file in the `mgsm`\ndirectory of the repository. Reference answers are public in the same files used for evaluation, so\nthere is no held-out grading server.\n\n## Who publishes it\n\nMGSM was introduced by Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush\nVosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das and Jason Wei, a team\nspanning Google Research, Google Brain and academic co-authors, in \"Language Models are Multilingual\nChain-of-Thought Reasoners\" (arXiv 2210.03057, posted October 2022). No formal conference publication\nwas confirmed for this page; the paper is tracked here as an arXiv preprint. The dataset and\nevaluation code are maintained in Google Research's `url-nlp` GitHub repository, which also hosts\nseveral unrelated under-resourced-language datasets. No dedicated MGSM leaderboard site was found;\naggregators such as llm-stats.com track scores that vendors report in their own model releases.\n\n## Lineage\n\nMGSM translates the 250-problem evaluation set built from GSM8K (Cobbe et al., 2021; `gsm8k` in this\nrepository) into ten additional languages. It has no announced successor and the repository lists no\nfurther language variants beyond the original ten plus English. It sits alongside, rather than\nsupersedes, English-only GSM8K: models are commonly reported on both.\n\n## Saturation and contamination\n\nMGSM shows the signature of a benchmark under pressure at the top without being fully saturated.\nAggregator llm-stats.com (accessed September 2026) lists Llama 4 Maverick at 92.3%, o3-mini at 92.0%\nand Claude 3.5 Sonnet at 91.6% as the top three of 31 tracked models, a gap of well under a point\nbetween the leaders, while the field average sits at 77.9%. That combination \u2014 a compressed top end\nover a still-wide overall spread \u2014 is why this page marks it \"watch\" rather than \"open\" or fully\n\"saturated.\" Contamination risk is high: both the problems and their answers are public in plain text\nand have been since October 2022, and the underlying GSM8K English problems have circulated since\n2021 and are widely believed to be present in pretraining corpora; translated or paraphrased versions\nof the same problems are a plausible path for that contamination to reach non-English training data\nas well.\n\n## How to run it\n\nlm-evaluation-harness implements MGSM as the task groups `mgsm_direct` and `mgsm_cot_native`, each\nwith one task per language (for example `mgsm_direct_en`, `mgsm_cot_native_de`). The reference data\nand few-shot exemplars live in Google Research's `url-nlp` repository under `mgsm/`. Because the\nharness supports both a no-reasoning and a native-language chain-of-thought protocol, and vendors\nsometimes report an English-chain-of-thought variant instead, check which protocol and which subset\nof languages (all eleven, or just the ten translations) a reported average covers before treating two\nMGSM scores as comparable.\n\n## Reading the numbers\n\nA strong MGSM average means a model's arithmetic word-problem reasoning survives translation into\ntypologically distant languages, including lower-resource ones like Swahili and Bengali; a weak one\nusually reflects English-centric training rather than an inability to do the arithmetic itself. The\noverall average can hide large per-language gaps, so check the per-language breakdown, not just the\nmean, before concluding a model is multilingual-capable. Given the high contamination risk and the\ncompressed spread at the top of the field, treat small differences between frontier models' MGSM\naverages as noise, and weight it alongside a genuinely held-out or recent multilingual reasoning\nbenchmark where one is available.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mgsm",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Cerebras-Llama-4-Maverick-17B-128E-Instruct",
   "model_id": "meta/cerebras-llama-4-maverick-17b-128e-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 92.3,
   "source": "llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Maverick 17B 128E Instruct",
   "model_id": "meta/llama-4-maverick-17b-128e-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 92.3,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-4-Maverick-17B-128E-Instruct-FP8",
   "model_id": "meta/llama-4-maverick-17b-128e-instruct-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 92.3,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o3-mini",
   "model_id": "openai/o3-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 92.0,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5",
   "model_id": "anthropic/claude-3-5-sonnet-20240620",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 91.6,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5 v2",
   "model_id": "anthropic/claude-3-5-sonnet-20241022",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 91.6,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.3 70B Instruct NVFP4",
   "model_id": "nvidia/llama-3-3-70b-instruct-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 91.1,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-3.3-70B-Instruct",
   "model_id": "meta/llama-3-3-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 91.1,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B Instruct",
   "model_id": "meta/meta-llama-3-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 91.1,
   "source": "llm-stats, open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B Instruct",
   "model_id": "nous-research/meta-llama-3-70b-instruct",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 91.1,
   "source": "llm-stats, open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o1-preview",
   "model_id": "openai/o1-preview",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 90.8,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 3",
   "model_id": "anthropic/claude-3-opus-20240229",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 90.7,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Scout 17B 16E",
   "model_id": "meta/llama-4-scout-17b-16e",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 90.6,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Scout 17B 16E Instruct",
   "model_id": "meta/llama-4-scout-17b-16e-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 90.6,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-4-Scout-17B-16E-Instruct-FP8",
   "model_id": "meta/llama-4-scout-17b-16e-instruct-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 90.6,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o",
   "model_id": "openai/gpt-4o",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 90.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-05-13)",
   "model_id": "openai/gpt-4o-2024-05-13",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 90.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-08-06)",
   "model_id": "openai/gpt-4o-2024-08-06",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 90.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-11-20)",
   "model_id": "openai/gpt-4o-2024-11-20",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 90.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o1",
   "model_id": "openai/o1",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 89.3,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o1-mini",
   "model_id": "openai/o1-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 89.3,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o1-pro",
   "model_id": "openai/o1-pro",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 89.3,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4 Turbo",
   "model_id": "openai/gpt-4-turbo",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 88.5,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Pro",
   "model_id": "google/gemini-1-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro",
   "model_id": "google/gemini-2-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o mini",
   "model_id": "openai/gpt-4o-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 87.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 90B Vision Instruct",
   "model_id": "meta/llama-3-2-90b-vision-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 86.9,
   "source": "llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3.5",
   "model_id": "anthropic/claude-3-5-haiku-20241022",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 85.6,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3.5 (latest)",
   "model_id": "anthropic/claude-3-5-haiku-latest",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 85.6,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3",
   "model_id": "anthropic/claude-3-sonnet-20240229",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 83.5,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 235B-A22B",
   "model_id": "qwen/qwen3-235b-a22b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 83.5,
   "source": "lmarena.ai, provider-reports, preference-evals, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Flash",
   "model_id": "google/gemini-1-5-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 82.6,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Flash-8B",
   "model_id": "google/gemini-1-5-flash-8b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 82.6,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.0 Flash",
   "model_id": "google/gemini-2-0-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 82.6,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash",
   "model_id": "google/gemini-2-5-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 82.6,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "phi 4",
   "model_id": "microsoft/phi-4",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 80.6,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2,, domain-evals llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3",
   "model_id": "anthropic/claude-3-haiku-20240307",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 75.1,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4",
   "model_id": "openai/gpt-4",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 74.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 11B Vision",
   "model_id": "meta/llama-3-2-11b-vision",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 68.9,
   "source": "llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3n 4B",
   "model_id": "google/gemma-3n-e4b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 67.0,
   "source": "llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Phi 4 mini instruct",
   "model_id": "microsoft/phi-4-mini-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 63.9,
   "source": "lmarena.ai, provider-reports, preference-evals, llm-stats, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 3B",
   "model_id": "meta/llama-3-2-3b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 58.2,
   "source": "llm-stats, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 3B Instruct",
   "model_id": "meta/llama-3-2-3b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 58.2,
   "source": "llm-stats, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-3.5-turbo",
   "model_id": "openai/gpt-3-5-turbo",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 56.3,
   "source": "lmarena.ai, provider-reports, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3.5 mini instruct",
   "model_id": "microsoft/phi-3-5-mini-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 47.9,
   "source": "open-llm-leaderboard-v2, llm-stats"
  }
 ],
 "page": {
  "aliases": [
   "Multilingual Grade School Math Benchmark",
   "Multilingual GSM8K"
  ],
  "category": "math",
  "contamination": {
   "note": "Problems and answers are public in plain-text TSV files and have been on GitHub since October 2022. The underlying GSM8K English problems have been public since 2021 and are widely believed to appear in pretraining corpora; translated or paraphrased copies of those same problems are a plausible route for the same contamination to reach non-English training data.",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en",
    "es",
    "fr",
    "de",
    "ru",
    "zh",
    "ja",
    "th",
    "sw",
    "bn",
    "te"
   ],
   "license": "CC-BY-4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 250,
   "size_note": "250 source problems, each translated into 10 languages; the repository ships 11 files of 250 rows each (10 translations plus the original English GSM8K subset).",
   "splits": "no train/test split; 250 evaluation problems per language, with separately translated few-shot exemplars in exemplars.py",
   "url": "https://github.com/google-research/url-nlp/tree/main/mgsm"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice G",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "mgsm_direct, mgsm_cot_native (per-language variants, e.g. mgsm_direct_en, mgsm_cot_native_de)",
   "opencompass": "",
   "other": ""
  },
  "id": "mgsm",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "gsm8k",
   "successors": [],
   "variants": []
  },
  "measures": "MGSM gives a model the same 250 grade-school arithmetic word problems used in GSM8K, each professionally translated by human annotators into ten languages (Spanish, French, German, Russian, Chinese, Japanese, Thai, Swahili, Bengali, Telugu), alongside the English originals. The model reads one problem in one language and produces a final numeric answer, typically after a worked chain-of-thought solution. The paper frames this as a test of multilingual chain-of-thought reasoning, not of translation quality: the translation is fixed and given to the model, and what is scored is whether multi-step arithmetic reasoning still works once the problem is not in English.\n",
  "metric": {
   "baseline_note": "The source paper reports no human baseline for MGSM. Scores are usually reported per language and then averaged; direct vs chain-of-thought and native vs English exemplar prompting change results substantially, so protocol must match before comparing two numbers.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy (exact match on final numeric answer)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "MGSM (Multilingual Grade School Math)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2210.03057",
   "title": "Language Models are Multilingual Chain-of-Thought Reasoners",
   "url": "https://arxiv.org/abs/2210.03057",
   "year": 2022
  },
  "publisher": {
   "authors": [
    "Freda Shi",
    "Mirac Suzgun",
    "Markus Freitag",
    "Xuezhi Wang",
    "Suraj Srivats",
    "Soroush Vosoughi",
    "Hyung Won Chung",
    "Yi Tay",
    "Sebastian Ruder",
    "Denny Zhou",
    "Dipanjan Das",
    "Jason Wei"
   ],
   "org": "Google Research",
   "url": "https://github.com/google-research/url-nlp"
  },
  "released": "2022-10",
  "repo_url": "https://github.com/google-research/url-nlp/tree/main/mgsm",
  "saturation": {
   "as_of": "2026-09",
   "note": "Aggregator llm-stats.com lists Llama 4 Maverick at 92.3%, o3-mini at 92.0% and Claude 3.5 Sonnet at 91.6% across 31 tracked models, with a field average of 77.9%. The top of the field has compressed into a narrow band even though scores overall still spread widely, which is why this is watch rather than fully open or fully saturated.",
   "status": "watch",
   "top_score": 92.3
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "Language Models are Multilingual Chain-of-Thought Reasoners (Shi et al., 2022)",
    "url": "https://arxiv.org/abs/2210.03057"
   },
   {
    "accessed": "2026-09-07",
    "title": "Language Models are Multilingual Chain-of-Thought Reasoners (ar5iv HTML rendering)",
    "url": "https://ar5iv.labs.arxiv.org/html/2210.03057"
   },
   {
    "accessed": "2026-09-07",
    "title": "google-research/url-nlp, mgsm directory (data, exemplars, licence, README)",
    "url": "https://github.com/google-research/url-nlp/tree/main/mgsm"
   },
   {
    "accessed": "2026-09-07",
    "title": "Training Verifiers to Solve Math Word Problems (Cobbe et al., 2021) - introduces GSM8K",
    "url": "https://arxiv.org/abs/2110.14168"
   },
   {
    "accessed": "2026-09-07",
    "title": "lm-evaluation-harness mgsm task README (task group names)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/mgsm/README.md"
   },
   {
    "accessed": "2026-09-07",
    "title": "MGSM Leaderboard | llm-stats.com",
    "url": "https://llm-stats.com/benchmarks/mgsm"
   }
  ],
  "status": "active",
  "subcategory": "multilingual math reasoning",
  "summary": "The same 250 GSM8K grade-school math problems, human-translated into ten languages, to test whether chain-of-thought reasoning holds up outside English.",
  "tags": [
   "math",
   "multilingual",
   "chain-of-thought",
   "grade-school-math"
  ],
  "task_format": "Free-response grade-school word problem in one of eleven languages; the model outputs a final answer as an Arabic numeral, usually after a chain-of-thought solution."
 }
}