OpenCompass's Chinese-instruction variant of HumanEval: the same 164 Python problems, evaluated with the task instruction given in Chinese rather than English.
unassessed
| Category | coding |
|---|---|
| Subcategory | function-level code generation (Chinese-language prompt) |
| Page status | active |
| Metric | pass@1 |
| Direction | higher_is_better |
| Unit | % |
| Publisher | OpenCompass Contributors |
humaneval_cn evaluates the same short, self-contained Python function-completion task as HumanEval, with the instruction given in Chinese instead of English. OpenCompass's configuration wraps each problem with a Chinese-language instruction, "完成以下Python代码任务:" ("Complete the following Python code task:"), ahead of the problem's own prompt text, and loads that prompt text from a separate dataset repository (opencompass/humaneval_cn) rather than the English-language opencompass/humaneval repository used for the plain HumanEval and HumanEval+ configs. Using a distinct dataset rather than just swapping the instruction text suggests the docstrings themselves are also translated into Chinese, not only the wrapper instruction, though this could not be directly confirmed: the dataset repository requires Hugging Face authentication to open, which this research could not obtain.
Complete a Python function body from a signature and docstring presented behind a Chinese-language instruction wrapper (and, per the separate dataset used, plausibly Chinese-translated docstrings, though this is not independently confirmed); graded by executing the completion against HumanEval's standard unit tests (pass@k), not the expanded EvalPlus tests.
No model card in ModelSpec reports this benchmark yet.