MultiPL-E: Python

The Python subset of MultiPL-E: the original, untranslated HumanEval and MBPP problems, used as the harness's reference language.

unassessed

What it measures

This subset is Python itself: the original HumanEval and MBPP function-completion problems that every other MultiPL-E language is translated from, run through the same execution harness as the translated languages. It exists so a model's in-language Python score can be compared directly against its scores on the translated languages, using one consistent harness and prompt style.

Task format

Function completion in Python: given the original signature, docstring and doctests, the model generates a function body, which is executed and checked against the original unit tests.

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub