MultiPL-E

MultiPL-E mechanically translates the HumanEval and MBPP Python code-generation benchmarks into 18+ other programming languages and scores pass@k in each.

unassessed

What it measures

MultiPL-E tests whether a code model's ability to solve short, self-contained function-completion problems in Python carries over to other programming languages. It takes the same underlying problems used by HumanEval (164 hand-written tasks) and MBPP (a larger set of crowd-written "basic Python" tasks) and mechanically translates each problem's signature, docstring and tests into a target language, leaving the algorithmic content unchanged. The model is given a function signature and docstring in the target language and must produce a working function body.

Task format

Function completion: given a translated signature, docstring and (for HumanEval-derived items) doctests, the model generates a function body. The completion is inserted into a per-language test harness, compiled or interpreted with the language's real toolchain inside a container, and run against translated unit tests.

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub