{
 "body": "\n## What it measures\n\nhumaneval_cn evaluates the same short, self-contained Python function-completion task as HumanEval,\nwith the instruction given in Chinese instead of English. OpenCompass's configuration wraps each\nproblem with a Chinese-language instruction, \"\u5b8c\u6210\u4ee5\u4e0bPython\u4ee3\u7801\u4efb\u52a1:\" (\"Complete the following Python\ncode task:\"), ahead of the problem's own prompt text, and loads that prompt text from a separate\ndataset repository (opencompass/humaneval_cn) rather than the English-language opencompass/humaneval\nrepository used for the plain HumanEval and HumanEval+ configs. Using a distinct dataset rather than\njust swapping the instruction text suggests the docstrings themselves are also translated into\nChinese, not only the surrounding instruction, though this could not be directly confirmed: the\ndataset repository requires Hugging Face authentication to open, which this research could not obtain.\n\n## How it is scored\n\nScoring follows OpenCompass's standard HumanEvalEvaluator, the same execution-based pass@k checker\nused for plain HumanEval and not the expanded HumanEval+ test suite: a completion is generated from\nthe Chinese-wrapped prompt, and the OpenCompass config requests k=[1, 10, 100], though most reporters\ncompute only pass@1 from a single greedy sample. Because the harness reuses the human-eval Python\nexecution checker, the underlying correctness tests are presumably still the original,\nEnglish-authored HumanEval unit tests, applied to a model's response to a Chinese-language prompt --\nso this benchmark most directly measures whether a model can follow a Chinese instruction and still\nproduce correct Python, and only tests genuinely Chinese-native problem comprehension if the\ndocstrings themselves are translated too.\n\n## Dataset and licence\n\nNo licence is stated for opencompass/humaneval_cn in any source this research could open, and its\nsize could not be independently confirmed: the Hugging Face repository\n(huggingface.co/datasets/opencompass/humaneval_cn) returns an authentication error rather than a\npublic dataset card, and no independent mirror was found. The OpenCompass loader code\n(`HumanevalDataset`) is the same one used for plain HumanEval, which is a 164-problem, single-split\nJSONL format, so 164 problems is a reasonable expectation, but this is not established from a source\nthis research could read.\n\n## Who publishes it\n\nNo paper specific to this Chinese-language translation of HumanEval was found. The configuration is\nmaintained inside the OpenCompass project, self-credited in its own repository as \"OpenCompass\nContributors\" (github.com/open-compass/opencompass), with no individual authors identified for this\nspecific dataset or its translation. OpenCompass's own citation is a general framework reference\n(\"OpenCompass: A Universal Evaluation Platform for Foundation Models,\" 2023) that does not describe\nhumaneval_cn specifically.\n\n## Lineage\n\nhumaneval_cn's predecessor is HumanEval (this repository's humaneval page): it is a same-problem,\ntranslated-prompt variant of the same 164 problems, run through OpenCompass's standard, non-Plus\nevaluator. It is unrelated to humanevalx (CodeGeeX's HumanEval-X, about programming-language\ncoverage) and to humaneval_multi (OpenCompass's own name for MultiPL-E's HumanEval-derived\ntranslations, also about programming-language coverage) -- humaneval_cn is the only one of this\nrepository's HumanEval descendants that varies the natural language of the instruction rather than\nthe target programming language. No successor to humaneval_cn was found.\n\n## Saturation and contamination\n\nNo public leaderboard specifically tracking humaneval_cn scores was found during this research, so\nsaturation is graded \"unknown\" rather than guessed. Contamination risk is high: whatever the exact\nscope of translation, the underlying task is HumanEval's own 164 problems and canonical solutions,\npublic since July 2021, so any model that has memorised HumanEval carries the same advantage here, on\ntop of whatever additional exposure Chinese-language copies or discussions of HumanEval may have\nseparately accumulated online.\n\n## How to run it\n\nThe OpenCompass config directory is `humaneval_cn` (several generations of config files exist, for\nexample `humaneval_cn_gen_6313aa.py`), but the dataset it registers reports under the column name\n`openai_humaneval_cn`, not `humaneval_cn` -- check for that name, not the directory name, when reading\nOpenCompass result tables. Running it requires the same `human_eval` package (and its\ndeliberately-disabled-by-default execution call) as plain HumanEval, plus access to the gated\nopencompass/humaneval_cn dataset.\n\n## Reading the numbers\n\nBecause this benchmark could not be independently confirmed to translate the docstrings rather than\nonly the wrapper instruction, treat a reported humaneval_cn score cautiously: a low score could\nreflect either weaker Chinese instruction-following or weaker coding ability, and no public\ndocumentation this research could locate distinguishes the two. It shares plain HumanEval's other\nlimits -- short, self-contained Python functions, a problem set that has been public for years --\nwithout a harder or refreshed test suite to offset them. Prefer a more clearly and openly documented\nbenchmark when comparing models' non-English coding ability, and treat any humaneval_cn number as\nprovisional until OpenCompass documents the dataset's construction more fully.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "humaneval_cn",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "openai_humaneval_cn"
  ],
  "category": "coding",
  "contamination": {
   "note": "Whatever the exact scope of translation, the underlying task is HumanEval's own 164 problems and canonical solutions, public since July 2021, so any model that has memorised HumanEval carries the same advantage here, on top of whatever additional exposure Chinese-language copies or discussions of HumanEval may have separately accumulated online.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "Chinese"
   ],
   "license": "",
   "modalities": [
    "text",
    "code"
   ],
   "public_test_set": null,
   "size": null,
   "size_note": "Not established. The OpenCompass loader (`HumanevalDataset`) is the same class used for plain HumanEval, which is a 164-problem, single-split JSONL format, so 164 problems is a reasonable expectation, but the dataset repository itself (huggingface.co/datasets/opencompass/humaneval_cn) returns an authentication error rather than a public dataset card or file listing, and no independent mirror was found, so the size could not be confirmed by counting.\n",
   "splits": "presumably a single test split mirroring HumanEval's; not independently confirmed",
   "url": "https://huggingface.co/datasets/opencompass/humaneval_cn"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 6, slice A",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "openai_humaneval_cn",
   "other": "The OpenCompass config directory is named humaneval_cn, but the dataset it registers reports under the column name openai_humaneval_cn, not humaneval_cn -- the two names refer to the same configuration."
  },
  "id": "humaneval_cn",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "humaneval",
   "predecessor": "humaneval",
   "successors": [],
   "variants": []
  },
  "measures": "humaneval_cn evaluates the same short, self-contained Python function-completion task as HumanEval, with the instruction given in Chinese instead of English. OpenCompass's configuration wraps each problem with a Chinese-language instruction, \"\u5b8c\u6210\u4ee5\u4e0bPython\u4ee3\u7801\u4efb\u52a1:\" (\"Complete the following Python code task:\"), ahead of the problem's own prompt text, and loads that prompt text from a separate dataset repository (opencompass/humaneval_cn) rather than the English-language opencompass/humaneval repository used for the plain HumanEval and HumanEval+ configs. Using a distinct dataset rather than just swapping the instruction text suggests the docstrings themselves are also translated into Chinese, not only the wrapper instruction, though this could not be directly confirmed: the dataset repository requires Hugging Face authentication to open, which this research could not obtain.\n",
  "metric": {
   "baseline_note": "Scored with OpenCompass's standard HumanEvalEvaluator -- the same execution-based pass@k checker used for plain HumanEval, not the stricter HumanEvalPlusEvaluator used for humaneval_plus. The OpenCompass config requests k=[1, 10, 100], though most reporters compute only a single greedy-sampled pass@1. No random-guess or human baseline is established.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "pass@1",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "HumanEval-CN",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [],
   "org": "OpenCompass Contributors",
   "url": "https://github.com/open-compass/opencompass"
  },
  "released": "",
  "repo_url": "https://github.com/open-compass/opencompass",
  "saturation": {
   "as_of": "",
   "note": "No public leaderboard specifically tracking humaneval_cn scores was found during this research (OpenCompass's own CompassRank site was not checked for a matching column), so saturation is graded \"unknown\" rather than guessed.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "open-compass/opencompass repository",
    "url": "https://github.com/open-compass/opencompass"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass humaneval_cn config",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/humaneval_cn/humaneval_cn_gen_6313aa.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass HumanevalDataset / HumanEvalEvaluator source",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/datasets/humaneval.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "opencompass/humaneval_cn dataset page (requires authentication; gated or private, size and card not readable)",
    "url": "https://huggingface.co/datasets/opencompass/humaneval_cn"
   }
  ],
  "status": "active",
  "subcategory": "function-level code generation (Chinese-language prompt)",
  "summary": "OpenCompass's Chinese-instruction variant of HumanEval: the same 164 Python problems, evaluated with the task instruction given in Chinese rather than English.",
  "tags": [
   "code-generation",
   "python",
   "chinese",
   "pass-at-k",
   "functional-correctness",
   "translated-prompt"
  ],
  "task_format": "Complete a Python function body from a signature and docstring presented behind a Chinese-language instruction wrapper (and, per the separate dataset used, plausibly Chinese-translated docstrings, though this is not independently confirmed); graded by executing the completion against HumanEval's standard unit tests (pass@k), not the expanded EvalPlus tests.\n"
 }
}