MultiPL-E: C++

The C++ subset of MultiPL-E: HumanEval and MBPP function-completion problems translated into C++ and scored with pass@1.

unassessed

What it measures

This subset translates the HumanEval and MBPP prompts into C++ by rewriting each problem's function signature, docstring and tests with C++ syntax and typing, then asks the model to complete the function body in C++. The underlying algorithmic problem is unchanged from the Python original; only the surface language differs.

Task format

Function completion in C++: given a translated signature, docstring and (for HumanEval-derived items) doctests, the model generates a function body, which is compiled or interpreted with a real C++ toolchain inside a container and checked against translated unit tests.

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub