MultiPL-E: Java

The Java subset of MultiPL-E: HumanEval and MBPP function-completion problems translated into Java and scored with pass@1.

unassessed

What it measures

This subset translates the HumanEval and MBPP prompts into Java by rewriting each problem's function signature, docstring and tests with Java syntax and typing, then asks the model to complete the function body in Java. The underlying algorithmic problem is unchanged from the Python original; only the surface language differs.

Task format

Function completion in Java: given a translated signature, docstring and (for HumanEval-derived items) doctests, the model generates a function body, which is compiled or interpreted with a real Java toolchain inside a container and checked against translated unit tests.

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub