CodeGeeX's multilingual HumanEval: 820 hand-crafted problems across Python, C++, Java, JavaScript and Go, extending the same 164 tasks by hand rather than by machine translation.
unassessed
| Category | coding |
|---|---|
| Subcategory | multilingual code generation |
| Page status | active |
| Metric | pass@1 |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 820 |
| Dataset licence | Apache-2.0 |
| Publisher | Tsinghua University; Zhipu.AI; Huawei |
HumanEval-X measures whether a code model's ability transfers beyond Python by hand-crafting the same underlying problems in four more languages. For each of HumanEval's 164 tasks, CodeGeeX's authors wrote an equivalent declaration, docstring, canonical solution and test suite in C++, Java, JavaScript and Go, alongside the existing Python version, for 820 problems in total. Two tasks are supported: code generation, where the model sees a declaration and docstring and must produce the solution, and code translation, where the model sees declarations in two languages plus a solution in one and must produce the equivalent solution in the other.
Given a function declaration and docstring in the target language, generate the function body (code generation); or given declarations in two languages and a solution in the source language, generate the equivalent solution in the target language, with the natural-language description removed (code translation). Both are graded by executing against per-language unit tests (pass@k).
No model card in ModelSpec reports this benchmark yet.