CodeEval-Pro's harder MBPP: each base problem is paired with a second task the model must solve by calling its own solution to the first.
unassessed
| Category | coding |
|---|---|
| Subcategory | self-invoking Python generation built on MBPP problems |
| Page status | active |
| Metric | pass@1 |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 378 |
| Dataset licence | MIT |
| Publisher | Tsinghua University; Yale University |
MBPP Pro is the MBPP half of HumanEval Pro and MBPP Pro (Yu et al., 2024). Each item gives a base MBPP-style problem and a related, harder problem in one prompt. The model must emit working Python for both, and the second solution is meant to call the first rather than reimplement it. The paper calls this self-invoking code generation: composition, not isolated snippets. Hugging Face CodeEval-Pro/mbpp-pro holds 378 such pairs.
Zero-shot (default) generation of two Python solutions in one response, from raw_problem and new_problem fields; graded by executing both against author-reviewed tests (pass@1). The repo also defines mbpp_pro_cot and mbpp_pro_1shot variants.
No model card in ModelSpec reports this benchmark yet.