MMMLU-lite

A 19,950-item multilingual MMLU slice with 25 test questions per subject-language pair across 57 subjects and 14 languages.

Also known as: MMMLU Lite, Global MMLU lite

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryknowledge
Subcategorymultilingual four-choice MMLU slice in 14 language varieties
Page statusactive
Metricaccuracy (OpenCompass AccwithDetailsEvaluator)
Directionhigher_is_better
Unit%
Dataset size19950
Dataset licenceMIT
PublisherOpenCompass

What it measures

MMMLU-lite evaluates broad academic and professional knowledge in translated MMLU questions. It covers 57 subjects and 14 language varieties, including Arabic, Bengali, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Brazilian Portuguese, Swahili, Yoruba, and Chinese. Each prompt is text-only.

Task format

Four-choice multiple choice. OpenCompass uses language-specific prompts and zero-shot retrieval, then extracts the first A/B/C/D option. The dataset is a compact fixed test slice rather than a separately reported training benchmark.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub