Okapi multilingual MMLU

lm-eval tag m_mmlu: ChatGPT-translated MMLU in 34 language configs, four-choice accuracy on the test split.

Also known as: m_mmlu, mmlu_multilingual, alexandrainst/m_mmlu

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	knowledge
Subcategory	machine-translated four-choice academic knowledge
Page status	active
Metric	accuracy
Direction	higher_is_better
Unit	%
Dataset size	13258
Dataset licence	CC-BY-NC-4.0
Publisher	University of Oregon NLP (Okapi translations); Alexandra Institute (Hub dump); EleutherAI (lm-eval tasks)

What it measures

okapi_mmlu_multilingual is EleutherAI's wrap of Alexandra Institute's alexandrainst/m_mmlu. Each item is a four-option academic question in one language, flattened from English [MMLU](mmlu.md) subjects. The model must pick A–D. University of Oregon translated most languages with GPT-3.5-turbo for Okapi; Icelandic used Greynir and Norwegian used DeepL. This is not a native-exam suite such as [ArabicMMLU](arabic_mmlu.md) or [GreekMMLU](greekmmlu.md), and it is not [Global-MMLU](global_mmlu.md).

Task format

Four-option multiple choice. Prompt is instruction plus A–D options and "Answer:". Target is the answer letter. fewshot_split is train with sampler first_n; test_split is test. Runnable tasks are m_mmlu_{lang} for 34 Hub configs. YAML tag is m_mmlu. There is no README in this harness folder.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub