A harder successor to HumanEval that pairs each of its 164 problems with a second, more complex problem the model must solve by correctly invoking its own solution to the first.
unassessed
| Category | coding |
|---|---|
| Subcategory | self-invoking code generation |
| Page status | active |
| Metric | pass@1 |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 164 |
| Dataset licence | MIT |
| Publisher | Tsinghua University; Yale University |
HumanEval Pro pairs each of HumanEval's 164 problems with a second, harder problem designed to be solved by calling the first problem's own solution. A model is given both problems in one prompt and must produce working Python for each, with the second implementation expected to invoke the first rather than reimplement its logic from scratch. The paper calls this "self-invoking code generation": it tests whether a model that can already write a correct function can also compose that function into something more complex, closer to how real code gets built than one isolated function at a time.
Given a base problem (one of HumanEval's 164) and a related, harder problem in the same prompt, generate Python solutions to both in one response, where the second solution is meant to call the first; graded by executing both against test suites the authors built and manually reviewed (pass@1).
No model card in ModelSpec reports this benchmark yet.