{
 "body": "\n## What it measures\n\nMMMLU-lite is a multilingual version of the MMLU knowledge exam. It asks four-choice academic and professional questions in 14 language varieties. The 57 subjects range from elementary mathematics and history to law, medicine, physics, and computer science.\n\nThe lite release samples a small fixed set for practical evaluation. It measures subject knowledge through translated prompts and also exposes language-specific prompt handling. It does not isolate translation quality from subject knowledge.\n\n## How it is scored\n\nOpenCompass runs each language configuration zero-shot and extracts the first A, B, C, or D token. Accuracy is the fraction of selected letters matching the target. Uniform-choice chance is 25%. Scores should be reported per language and subject before any cross-language average.\n\n## Dataset and licence\n\nThe public release contains 14 language varieties and 57 subjects, with 25 test questions in each subject-language cell: 19,950 questions in total. The Hub metadata labels it MIT. Each configuration is a test JSONL file, so the evaluation answers are public.\n\n## Who publishes it\n\nThe dataset is distributed by the OpenCompass project. The OpenCompass configuration names the language varieties and builds one zero-shot dataset entry for each. The upstream MMMLU dataset is the broader multilingual MMLU family; this page documents the OpenCompass lite slice.\n\n## Lineage\n\nMMMLU-lite is a compact variant of [MMMLU](mmmlu.md), which in turn extends [MMLU](mmlu.md) across languages. It is not the English MMLU family and should not be merged with translated full-size MMMLU results.\n\n## Saturation and contamination\n\nA current saturation status is not established for this slice. The source questions derive from public MMLU material and the translated test files are public. Contamination risk is therefore high, especially for models trained on multilingual web and benchmark collections.\n\n## How to run it\n\nUse OpenCompass\u2019s `mmmlu_lite` configuration. It loads `opencompass/mmmlu_lite`, reads the `test` JSONL for each language directory, formats the prompt in that language, and applies `first_option_postprocess` over A/B/C/D. Prompt language and aggregation must be stated.\n\n## Reading the numbers\n\nA high score means the model selected the keyed answer in a particular translated subject slice. It does not show that the model understood every wording nuance or that performance is comparable across languages. Compare per-subject results with English MMLU and full MMMLU under matched prompts.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mmmlu_lite",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "MMMLU Lite",
   "Global MMLU lite"
  ],
  "category": "knowledge",
  "contamination": {
   "note": "The test files are public translated MMLU material. The compact sampling does not establish protection from training contamination.",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "ar",
    "bn",
    "de",
    "es",
    "fr",
    "hi",
    "id",
    "it",
    "ja",
    "ko",
    "pt",
    "sw",
    "yo",
    "zh"
   ],
   "license": "MIT",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 19950,
   "size_note": "14 language varieties \u00d7 57 subjects \u00d7 25 test questions, as described by the public dataset release and configuration.",
   "splits": "One test JSONL per language variety; 25 examples per subject-language cell",
   "url": "https://huggingface.co/datasets/opencompass/mmmlu_lite"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna, luna-batch-059 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, luna-batch-059"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "mmmlu_lite",
   "other": ""
  },
  "id": "mmmlu_lite",
  "last_updated": "2024-11",
  "leaderboard_url": "",
  "lineage": {
   "family": "mmmlu",
   "predecessor": "mmmlu",
   "successors": [],
   "variants": []
  },
  "measures": "MMMLU-lite evaluates broad academic and professional knowledge in translated MMLU questions. It covers 57 subjects and 14 language varieties, including Arabic, Bengali, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Brazilian Portuguese, Swahili, Yoruba, and Chinese. Each prompt is text-only.\n",
  "metric": {
   "baseline_note": "All task configurations expose four choices, so uniform-choice chance is 25%; no human baseline was established.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy (OpenCompass AccwithDetailsEvaluator)",
   "random_baseline": 25.0,
   "unit": "%"
  },
  "name": "MMMLU-lite",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "MMMLU: Measuring Massive Multitask Language Understanding in Multiple Languages",
   "url": "https://huggingface.co/datasets/openai/MMMLU",
   "year": null
  },
  "publisher": {
   "authors": [],
   "org": "OpenCompass",
   "url": "https://github.com/open-compass/opencompass"
  },
  "released": "2024-10",
  "repo_url": "https://github.com/open-compass/opencompass",
  "saturation": {
   "as_of": "",
   "note": "No current multilingual leaderboard cell was established for this lite slice.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass MMMLU-lite dataset card",
    "url": "https://huggingface.co/datasets/opencompass/mmmlu_lite"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass MMMLU-lite configuration",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/mmmlu_lite/mmmlu_lite_gen_c51a84.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass MMMLULiteDataset loader",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/datasets/mmmlu.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face MMMLU-lite metadata",
    "url": "https://huggingface.co/api/datasets/opencompass/mmmlu_lite"
   }
  ],
  "status": "active",
  "subcategory": "multilingual four-choice MMLU slice in 14 language varieties",
  "summary": "A 19,950-item multilingual MMLU slice with 25 test questions per subject-language pair across 57 subjects and 14 languages.",
  "tags": [
   "multilingual",
   "mmlu",
   "knowledge",
   "multiple-choice",
   "translation"
  ],
  "task_format": "Four-choice multiple choice. OpenCompass uses language-specific prompts and zero-shot retrieval, then extracts the first A/B/C/D option. The dataset is a compact fixed test slice rather than a separately reported training benchmark.\n"
 }
}