MMLU's questions in 42 languages, with CS/CA labels on 2,850 items per language; lm-eval ships Lite (15 languages) and full groups.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | MMLU translated into 42 languages with cultural-sensitivity labels |
| Page status | active |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 14042 |
| Dataset licence | Apache-2.0 |
| Publisher | Cohere Labs (Cohere For AI), with EPFL, Hugging Face, Mila/McGill, AI Singapore, NUS, MIT, KAIST, and other collaborators named on the paper |
Global-MMLU tests whether MMLU-style four-choice academic knowledge holds after translation, and whether scores move when items need cultural, regional, or dialect knowledge. Each language config repeats the MMLU question set (14,042 test and 285 dev rows on Hugging Face, matching the cais/mmlu mirror used on the [MMLU](mmlu.md) page). A 2,850-question subset per language is labelled culturally sensitive (CS) or culturally agnostic (CA). The paper finds 28% of questions need culturally sensitive knowledge, and 84.9% of geographic items focus on North America or Europe. Translations mix professional post-edits (14 languages, including OpenAI MMMLU where available), community translations (11), and machine translation (16), plus English. This is not a native-exam suite such as [ArabicMMLU](arabic_mmlu.md) or [GreekMMLU](greekmmlu.md).
Four-option multiple choice (A–D). lm-eval prompts "{question}\\nA. ...\\nAnswer:" and scores accuracy on the letter. Lite groups (`global_mmlu_{lang}`) load CohereLabs/Global-MMLU-Lite; full groups (`global_mmlu_full_{lang}`) load CohereLabs/Global-MMLU. YAML sets fewshot_split to dev but does not pin a shot count.
No model card in ModelSpec reports this benchmark yet.