{
 "body": "\n## What it measures\n\nMBPP-CN asks a model to write a short Python function from a natural-language\ntask, with the instruction and the three few-shot exemplars in Chinese. The\nOpenCompass prompt is a professional-programmer wrapper such as\n\u300c\u4f60\u662f\u4e00\u540d\u4e13\u4e1a\u7684 Python \u7a0b\u5e8f\u5458\uff0c\u4f60\u7684\u4efb\u52a1\u662f\uff1a\u2026\u300d followed by the problem text\nand English `assert` tests, then a `[BEGIN]` / `[DONE]` completion format\ncopied from English MBPP. The target language of the code is still Python.\n\nWhether every test item's `{text}` field is a full Chinese translation, or\nonly the wrapper is Chinese, depends on `./data/mbpp_cn/mbpp_cn.jsonl`. That\nfile is loaded with `local_mode=True` and was not in the public sources\nopened here. Hugging Face `opencompass/mbpp_cn` returned HTTP 401.\n\n## How it is scored\n\nOpenCompass registers `MBPPEvaluator` without the `MBPPPlus` metric, so a\ntask counts as solved when the completion passes MBPP's original asserts,\nnot EvalPlus's larger suite. The reported figure is a percent pass rate\n(`score = pass / n * 100`). Deprecated configs in the same directory add\npass@k sampling and a ten-repeat variant; the live `mbpp_cn_gen.py` import\nis the greedy three-shot file `mbpp_cn_gen_9114d5.py`. Compare a number\nonly to another run that used the same config hash.\n\n## Dataset and licence\n\nNo licence or row count is stated in the config, the dataset-index, or a\npublic card this research could open. `MBPPDataset.load` maps JSON/HF input\nonto `train[:10]` and `train[10:510]`, which is 500 test tasks when the\nfile is ordered like original MBPP. That is a loader convention, not a\ncounted size of `mbpp_cn.jsonl`. The original English MBPP dump on Hugging Face\n(`google-research-datasets/mbpp`) is tagged CC BY 4.0. Whether the\nChinese file inherits that licence is not stated.\n\n## Who publishes it\n\nThere is no paper for MBPP-CN. OpenCompass's `dataset-index.yml` lists the\nname MBPP-CN under Code with an empty `paper` field. The configuration lives\nin the OpenCompass repository (github.com/open-compass/opencompass),\ncredited to OpenCompass Contributors. The parent English dataset is Google\nResearch's MBPP (Austin et al., arXiv 2108.07732, August 2021).\n\n## Lineage\n\nThe predecessor in this repository is [MBPP](mbpp.md). MBPP-CN is the\nnatural-language-instruction variant, analogous to\n[HumanEval-CN](humaneval_cn.md), not a new set of programming problems.\nIt is not [MBPP+](mbpp_plus.md) (stricter tests) or [MBPP Pro](mbpp_pro.md)\n(self-invoking pairs), and it is not MultiPL-E, which translates the target\nprogramming language rather than the prompt language.\n\n## Saturation and contamination\n\nSaturation is unknown: no isolated MBPP-CN leaderboard was found.\nContamination risk is high because the tasks are MBPP's, public with\nsolutions since 2021. A Chinese instruction does not make those solutions\nprivate.\n\n## How to run it\n\nIn OpenCompass the runnable dataset abbreviation is `mbpp_cn`\n(`opencompass/configs/datasets/mbpp_cn/mbpp_cn_gen.py` \u2192\n`mbpp_cn_gen_9114d5.py`). You need a local `data/mbpp_cn/mbpp_cn.jsonl` and\nthe same execution-based evaluator as English MBPP. Do not mix this score\nwith `mbpp`, `mbpp_plus`, or `mbpp_pro` columns.\n\n## Reading the numbers\n\nA high MBPP-CN pass rate means the model can follow a Chinese coding\ninstruction and still emit Python that clears MBPP's short tests. It does\nnot by itself show Chinese-native problem understanding if only the wrapper\nwas translated, and it does not show robustness: the grader is the original\nthree-assert suite. Read it next to English [MBPP](mbpp.md) and, for\nstricter tests, [MBPP+](mbpp_plus.md).\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mbpp_cn",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "MBPP CN",
   "MBPP Chinese"
  ],
  "category": "coding",
  "contamination": {
   "note": "The underlying problems are MBPP's crowd-sourced Python tasks, public with reference solutions since August 2021. A Chinese wrap does not hide those solutions. Extra exposure from Chinese copies of MBPP is possible and was not measured.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "zh"
   ],
   "license": "",
   "modalities": [
    "text",
    "code"
   ],
   "public_test_set": null,
   "size": null,
   "size_note": "Not counted from the JSONL. The loader is MBPPDataset, which in Hugging Face / JSON mode takes split train[:10] as the few-shot pool and train[10:510] as the test split (500 items) when the file is ordered like original MBPP. That 500-item expectation is a property of the loader, not a row count of mbpp_cn.jsonl. huggingface.co/datasets/opencompass/mbpp_cn returned HTTP 401, so size, licence and whether the prompts are fully translated remain unconfirmed. OpenCompass's dataset-index lists MBPP-CN with an empty paper field.\n",
   "splits": "OpenCompass MBPPDataset train[:10] / train[10:510] convention if the JSONL matches original MBPP order; not independently confirmed",
   "url": ""
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-006 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-006"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "mbpp_cn",
   "other": "Directory opencompass/configs/datasets/mbpp_cn; current entry mbpp_cn_gen.py imports mbpp_cn_gen_9114d5.py (abbr mbpp_cn). Deprecated files: deprecated_mbpp_cn_gen_1d1481.py, deprecated_mbpp_cn_passk_gen_1d1481.py, deprecated_mbpp_cn_repeat10_gen_1d1481.py."
  },
  "id": "mbpp_cn",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "mbpp",
   "successors": [],
   "variants": [
    "humaneval_cn"
   ]
  },
  "measures": "mbpp_cn is OpenCompass's Chinese-language wrap of Mostly Basic Python Problems. The model still has to write a short Python function that passes hidden asserts, but the wrapper instruction and the three few-shot exemplars are written in Chinese. The current config loads a local JSONL at ./data/mbpp_cn/mbpp_cn.jsonl through the same MBPPDataset class used for English MBPP, so the {text} field is whatever that file stores \u2014 likely translated prompts, though the JSONL itself was not opened here. Unit tests in the few-shot block remain English assert statements. It is a single-turn, text-to-code task.\n",
  "metric": {
   "baseline_note": "Scored with OpenCompass's MBPPEvaluator (the original three-assert suite), not the EvalPlus MBPPPlus metric. No random-guess or human baseline is stated for this config. Deprecated pass@k and repeat-10 configs exist in the same directory.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "pass@1 (OpenCompass MBPPEvaluator score, percent of tasks whose tests all pass)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "MBPP-CN",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [],
   "org": "OpenCompass Contributors",
   "url": "https://github.com/open-compass/opencompass"
  },
  "released": "",
  "repo_url": "https://github.com/open-compass/opencompass",
  "saturation": {
   "as_of": "",
   "note": "No public leaderboard that isolates mbpp_cn was found. OpenCompass's dataset-statistics page lists MBPP-CN as a supported code dataset without scores.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass mbpp_cn_gen_9114d5.py (current MBPP-CN config)",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/mbpp_cn/mbpp_cn_gen_9114d5.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass mbpp_cn_gen.py re-export",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/mbpp_cn/mbpp_cn_gen.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass MBPPDataset / MBPPEvaluator source",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/datasets/mbpp.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass dataset-index.yml (MBPP-CN, empty paper field)",
    "url": "https://github.com/open-compass/opencompass/blob/main/dataset-index.yml"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass dataset statistics (lists MBPP-CN)",
    "url": "https://opencompass.readthedocs.io/en/latest/dataset_statistics.html"
   },
   {
    "accessed": "2026-09-08",
    "title": "Program Synthesis with Large Language Models (original MBPP)",
    "url": "https://arxiv.org/abs/2108.07732"
   },
   {
    "accessed": "2026-09-08",
    "title": "google-research-datasets/mbpp Hub API (license cc-by-4.0; 974 tasks across full splits)",
    "url": "https://huggingface.co/api/datasets/google-research-datasets/mbpp"
   }
  ],
  "status": "active",
  "subcategory": "entry-level Python function generation with Chinese-language instructions",
  "summary": "OpenCompass's Chinese-instruction variant of MBPP: the same unit-tested Python tasks, prompted in Chinese rather than English.",
  "tags": [
   "code-generation",
   "python",
   "chinese",
   "pass-at-k",
   "translated-prompt"
  ],
  "task_format": "Three-shot generation. Each turn is a Chinese expert-programmer instruction, a task description, and English assert tests; the model continues after a [BEGIN] delimiter and is graded by executing the completion.\n"
 }
}