MMLUArabic (AceGPT translated MMLU)

AceGPT's GPT-3.5-Turbo translation of English MMLU into Arabic across 57 subjects; not the native-exam ArabicMMLU suite.

Also known as: Arabic MMLU (AceGPT), acegpt_MMLUArabic

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	knowledge
Subcategory	machine-translated English MMLU in Arabic (57 subjects)
Page status	active
Metric	accuracy
Direction	higher_is_better
Unit	%
Publisher	FreedomIntelligence / Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen, and King Abdullah University of Science and Technology

What it measures

MMLUArabic, in the OpenCompass and AceGPT sense, is English [MMLU](mmlu.md) machine-translated into Arabic. Each item is still a four-option academic question on subjects such as abstract_algebra and professional_law. AceGPT used Turbo (GPT-3.5-Turbo) for the translation. It measures whether MMLU knowledge survives that translation, not whether a model knows Arabic-only school content. That native-exam job is [arabic_mmlu](arabic_mmlu.md).

Task format

Four-option multiple-choice in Arabic. OpenCompass reads per-subject CSVs with columns input, A, B, C, D, target and scores accuracy. Three configs: 5-shot generation with first_option_postprocess over ABCD; 5-shot perplexity over A–D; zero-shot generation. Task abbrs are acegpt_MMLUArabic_{subject}.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub