MGSM (Multilingual Grade School Math)

The same 250 GSM8K grade-school math problems, human-translated into ten languages, to test whether chain-of-thought reasoning holds up outside English.

Also known as: Multilingual Grade School Math Benchmark, Multilingual GSM8K

unassessed

What it measures

MGSM gives a model the same 250 grade-school arithmetic word problems used in GSM8K, each professionally translated by human annotators into ten languages (Spanish, French, German, Russian, Chinese, Japanese, Thai, Swahili, Bengali, Telugu), alongside the English originals. The model reads one problem in one language and produces a final numeric answer, typically after a worked chain-of-thought solution. The paper frames this as a test of multilingual chain-of-thought reasoning, not of translation quality: the translation is fixed and given to the model, and what is scored is whether multi-step arithmetic reasoning still works once the problem is not in English.

Task format

Free-response grade-school word problem in one of eleven languages; the model outputs a final answer as an Arabic numeral, usually after a chain-of-thought solution.

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub