{
 "body": "\n## What it measures\n\njapanese_leaderboard is not one benchmark but an lm-evaluation-harness task group: invoking it runs eight\nseparately-authored Japanese evaluation tasks and reports each one's own result, not a single blended score.\nFour tasks come from JGLUE, Japan's JGLUE general-language-understanding suite -- JCommonsenseQA\n(commonsense multiple-choice), JNLI (natural-language inference), JSQuAD (extractive reading\ncomprehension) and MARC-ja (binary review-sentiment classification). The other four are JAQKET v2\n(Wikipedia-grounded Japanese quiz question answering), the Japanese subset of MGSM (grade-school maths word\nproblems), the Japanese subset of XL-Sum (news summarisation) and the Japanese subset of XWinograd\n(commonsense pronoun resolution). Because these eight tasks measure different skills on different scales, a\nsingle number under this group's name does not exist, and treating any one component's score as\nrepresentative of \"Japanese ability\" overall would misread what the group actually reports.\n\n## How it is scored\n\nEach of the eight tasks is scored on its own terms. JCommonsenseQA, JNLI, MARC-ja and XWinograd compare\nlog-likelihoods over labelled multiple-choice options. JAQKET v2 and JSQuAD generate free text and score it\nby exact match. MGSM generates a worked solution and extracts a final numeric answer for accuracy scoring.\nXL-Sum generates a summary and scores it with ROUGE-2. Few-shot counts differ by task, from zero-shot\n(XWinograd) up to five-shot (MGSM), and this repository's `arabic_leaderboard_complete` page is the\nprecedent for how this kind of many-tasks-under-one-name page is handled here: as in that case, this page\ntreats the group as a bundle of separate measurements rather than pretending a single figure summarises it.\nUnlike the Arabic group, though, japanese_leaderboard's own harness configuration defines no combined\nmetric at all -- there is no size-weighted or unweighted aggregate here, only the eight independent results.\n\n## Dataset and licence\n\nNo single item count or licence applies. The eight component tasks are drawn from three separate source\nprojects with their own dataset structures: JGLUE (JCommonsenseQA, JNLI, JSQuAD, MARC-ja), JAQKET, and\nseparately-adapted multilingual resources for MGSM, XL-Sum and XWinograd's Japanese portions. Each carries\nits own licence, none summarised into one answer here, and its own split structure -- generally a training\nsplit used for few-shot exemplars and a validation or test split used for scoring.\n\n## Who publishes it\n\nThis task group is maintained inside EleutherAI's lm-evaluation-harness. Its README credits the prompt\ntemplates to Stability AI's Japanese fork of the harness (`Stability-AI/lm-evaluation-harness`, branch\n`jp-stable`). The eight underlying tasks have separate authorship: JAQKET was introduced by Suzuki et al.\nin 2020; JGLUE (JCommonsenseQA, JNLI, JSQuAD, MARC-ja) by Kurihara, Kawahara and Shibata; MGSM extends\nGSM8K (Cobbe et al.) to multiple languages via Shi et al.; XL-Sum is from Hasan et al.; and XWinograd's\nmultilingual construction comes from Muennighoff et al. and Tikhonov and Ryabinin. None of these five\nsource projects has its own page in this repository yet, though `gsm8k` (in this repository) documents the\nEnglish original that MGSM extends to Japanese and ten other languages.\n\n## Lineage\n\nNo predecessor or successor is tracked for this id, and none of its eight component task groups has its\nown page in this repository yet. As with `arabic_leaderboard_complete`, this page's job is to name what the\ngroup aggregates and be explicit that the aggregate is not one measurement -- and the comparison between\nthe two pages is itself informative: the Arabic group defines a weighted combined score across its 14\ncomponents, while this Japanese group defines none across its eight, despite both being named and structured\nsimilarly as harness task groups.\n\n## Saturation and contamination\n\nBecause no combined score exists, whether japanese_leaderboard is \"saturated\" is not a well-formed question\nfor the group as a whole; it would need to be asked separately of each component, and no current\nper-component standings were established from a source read for this page. Contamination risk is high:\nevery component dataset is public with answers and has been for two to six years by the date of this\nresearch -- JAQKET since 2020, the four JGLUE tasks since 2022, and MGSM, XL-Sum and XWinograd's Japanese\nportions since 2021-2022 -- all well before the training cutoff of any model current today. No held-out or\nrefreshed portion is described for any of the eight groups.\n\n## How to run it\n\n`lm_eval --tasks japanese_leaderboard` runs all eight component tasks in one invocation and reports each\none's own metric; any component can also be run alone by name (for example `lm_eval --tasks\nja_leaderboard_jsquad`). Because there is no aggregate_metric_list in the group's configuration, there is no\nsingle number to compare across models from this group name -- comparing models means comparing the same\nnamed component's score across runs, not a \"japanese_leaderboard score.\"\n\n## Reading the numbers\n\nThere is no single \"japanese_leaderboard score\" to read, and a report that presents one without naming\nwhich of the eight components it came from should be treated with suspicion. Read each component the way\nits own benchmark is normally read: JSQuAD and JAQKET v2 as extractive reading comprehension, JNLI and\nJCommonsenseQA and MARC-ja and XWinograd as their respective classification or commonsense tasks, MGSM as\ngrade-school arithmetic reasoning in Japanese, and XL-Sum as summarisation quality by a lexical-overlap\nmetric that does not capture fluency or faithfulness well on its own. Given every component's long public\navailability, a very high score on any one of them is at least as likely to reflect prior exposure as\ngenuine task skill.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "japanese_leaderboard",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "ja_leaderboard"
  ],
  "category": "composite",
  "contamination": {
   "note": "Every component dataset is public with answers and has been for two to six years by the date of this research: JAQKET since 2020, JGLUE (JCommonsenseQA, JNLI, JSQuAD, MARC-ja) since 2022, MGSM and XL-Sum since 2021-2022, and XWinograd's Japanese portion since around the same period -- all well before the training cutoff of any model current at the time of this research. No held-out or refreshed portion is described for any of the eight groups.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "ja"
   ],
   "license": "Varies by component dataset (JGLUE, JAQKET, MGSM, XL-Sum and XWinograd each carry their own terms); not established as a single licence for this page.",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "No single item count applies. This aggregates eight heterogeneous task groups drawn from three separate source projects -- JGLUE (four tasks), JAQKET, and separately-adapted multilingual sets for MGSM, XL-Sum and XWinograd -- each with its own train/validation/test structure. No source read for this page gives a combined item count across all eight groups.\n",
   "splits": "each component task draws its own validation or test split for evaluation and its own training split for few-shot exemplars; no single split description applies across all eight",
   "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/japanese_leaderboard"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 6, slice D",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "japanese_leaderboard",
   "opencompass": "",
   "other": "The eight component task names are ja_leaderboard_jaqket_v2, ja_leaderboard_jcommonsenseqa, ja_leaderboard_jnli, ja_leaderboard_jsquad, ja_leaderboard_marc_ja, ja_leaderboard_mgsm, ja_leaderboard_xlsum and ja_leaderboard_xwinograd, each independently runnable. The group's own `_ja_leaderboard.yaml` lists only the group name and this eight-task list, with no `aggregate_metric_list` block -- confirmed by reading the file directly and comparing it against `arabic_leaderboard_complete.yaml`, which does define one.\n"
  },
  "id": "japanese_leaderboard",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "japanese_leaderboard is not one benchmark but an lm-evaluation-harness task group: running it executes eight separately-authored Japanese evaluation tasks and reports each one's own metric, not a single blended score. Four come from JGLUE, Japan's general language-understanding benchmark suite (JCommonsenseQA, JNLI, JSQuAD, MARC-ja); the other four are JAQKET v2 (Wikipedia-grounded quiz question answering), the Japanese subset of MGSM (grade-school maths word problems), the Japanese subset of XL-Sum (news summarisation), and the Japanese subset of XWinograd (commonsense pronoun resolution). A model's results under this group name are eight separate numbers on different scales -- exact match, classification accuracy and ROUGE-2 among them -- not one measurement of \"Japanese ability.\"\n",
  "metric": {
   "baseline_note": "No single random baseline applies: JNLI is a three-way choice, MARC-ja is binary, JCommonsenseQA and XWinograd have their own separate option counts, and JAQKET v2, JSQuAD, MGSM and XL-Sum are generation tasks with no fixed-choice baseline at all. Unlike this repository's `arabic_leaderboard_complete` page, the harness's own group config for this task (`_ja_leaderboard.yaml`, read directly for this page) does not define an `aggregate_metric_list`, so no weighted or unweighted combined number is computed automatically the way one is for the Arabic group.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "no combined metric: each of the eight component tasks reports its own metric (exact_match, acc, or a custom ROUGE-2 aggregation) independently",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "Japanese Leaderboard (lm-evaluation-harness)",
  "page_kind": "family",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [],
   "org": "lm-evaluation-harness (EleutherAI); the task group's README credits its prompts to Stability AI's Japanese fork of the harness",
   "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/japanese_leaderboard"
  },
  "released": "",
  "repo_url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/japanese_leaderboard",
  "saturation": {
   "as_of": "",
   "note": "Because the harness computes no combined score for this group, \"saturated\" is not a well-defined question for japanese_leaderboard as a whole the way it can be for a single-metric benchmark; it would have to be asked separately of each of the eight components. No current per-component standings were established from a source read for this page.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "japanese_leaderboard README: task list, sources and citations",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/japanese_leaderboard/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "_ja_leaderboard.yaml: group definition, eight-task list, no aggregate_metric_list",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/japanese_leaderboard/_ja_leaderboard.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "arabic_leaderboard_complete.yaml: comparison point showing an explicit aggregate_metric_list",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/arabic_leaderboard_complete/arabic_leaderboard_complete.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "ja_leaderboard_xlsum.yaml: ROUGE-2 metric definition",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/japanese_leaderboard/ja_leaderboard_xlsum.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "ja_leaderboard_mgsm.yaml: 5-shot generation config",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/japanese_leaderboard/ja_leaderboard_mgsm.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "ja_leaderboard_xwinograd.yaml: zero-shot multiple-choice config",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/japanese_leaderboard/ja_leaderboard_xwinograd.yaml"
   }
  ],
  "status": "active",
  "subcategory": "lm-evaluation-harness task group running eight independently-authored Japanese NLP tasks, reported without a combined score",
  "summary": "japanese_leaderboard is an lm-evaluation-harness group running eight independent Japanese NLP tasks -- JAQKET, four JGLUE tasks, MGSM, XL-Sum and XWinograd -- with no combined score computed.",
  "tags": [
   "composite",
   "japanese",
   "leaderboard",
   "multiple-choice",
   "question-answering",
   "summarization",
   "math",
   "commonsense"
  ],
  "task_format": "A group of eight independently-formatted tasks. JCommonsenseQA, JNLI, MARC-ja and XWinograd are multiple-choice or classification tasks scored by comparing log-likelihoods over labelled options. JAQKET v2 and JSQuAD are extractive question answering scored by exact match against generated text. MGSM is a generated chain-of-reasoning math answer scored by extracting a final number. XL-Sum is free-form generated summarisation scored by ROUGE-2. Few-shot counts also vary by component, from zero (XWinograd) to five (MGSM); no single prompt format or shot count applies across the group.\n"
 }
}