{
 "body": "\n## What it measures\n\nMBPP Pro pairs a base Mostly Basic Python problem with a second, harder\nproblem that should be solved by calling the first solution. One prompt,\ntwo Python programs, with the second expected to invoke the first. Yu,\nZhao, Cohan and Zhang (arXiv 2412.21199) introduce this as self-invoking\ncode generation, alongside the HumanEval-sized sibling HumanEval Pro.\n\n## How it is scored\n\nDefault reporting is greedy pass@1 on the combined self-invoking task.\nThe authors generate companion problems, candidate solutions and tests\nwith DeepSeek-V2.5, execute them for ground-truth outputs, then iterate\nhuman review until canonical solutions hit 100% pass@1 (paper Table 1,\nthree rounds on MBPP Pro). CodeEval-Pro's harness sanitizes model output\nand can report both raw and sanitized pass@k; they take the higher.\nOpenCompass's `mbpp_pro` config instead posts completions to a hosted\nHugging Face Space evaluator, so those numbers are not guaranteed to\nmatch the authors' local runner. The OpenCompass directory README shows\nsmall gaps (for example 66 vs 65 pass@1 for Qwen2.5-Coder-7B-Instruct).\n\n## Dataset and licence\n\nCodeEval-Pro/mbpp-pro is MIT-licensed, 378 rows, Hub last modified\n2024-12-31. The split is named `train` on the Hub even though the file\nis the evaluation set. That 378 equals current [MBPP+](mbpp_plus.md)\nsize, not the original 974-problem MBPP dump. All prompts are English;\nthe code is Python. Tests travel with the dataset.\n\n## Who publishes it\n\nZhaojian Yu and Xiao-Ping Zhang (Tsinghua) with Yilun Zhao and Arman\nCohan (Yale). The paper was posted 31 December 2024. Code, leaderboard\nand Hugging Face organisation use the name CodeEval-Pro\n(github.com/CodeEval-Pro/CodeEval-Pro;\nanswers111.github.io/evalpro.github.io).\n\n## Lineage\n\nPredecessor: [MBPP](mbpp.md). Sibling in the same paper:\n[HumanEval Pro](humaneval_pro.md). BigCodeBench-Lite Pro is a third\nconstruction in that paper and has no page here. [MBPP+](mbpp_plus.md)\nis EvalPlus's extra-test grader, not this self-invoking task. Do not\nfold `mbpp_pro` into `mbpp`.\n\n## Saturation and contamination\n\nOn the CodeEval-Pro `data.csv` (2026-09-08), the highest greedy 0-shot\nMBPP Pro pass@1 is 71.4 (DeepseekCoder-V2-Instruct). DeepSeek-R1, top\non HumanEval Pro at 79.2 in the same file, is 68.8 here. Smaller models\nsit much lower, so the scale still separates systems. Contamination is\nmedium: old public bases, new 2024 companions.\n\n## How to run it\n\nAuthors: `python -m eval.inference --dataset mbpp_pro` then sanitize and\n`python -m harness`. OpenCompass: `mbpp_pro_gen_3dc067.py` (and a\nrepeat config), evaluator at\n`https://opencompass-multiple-evaluator.hf.space`. State which runner\nand whether CoT or 1-shot was used.\n\n## Reading the numbers\n\nA strong MBPP Pro score means the model can write a basic function and\nthen use that function inside a harder follow-up. The drop from MBPP or\nMBPP+ to MBPP Pro is the result the benchmark is built to show. It is\nstill Python-only, still public, and still not multi-file engineering.\nRead it with [HumanEval Pro](humaneval_pro.md) and a contamination-aware\nset such as [LiveCodeBench](live_code_bench.md).\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mbpp_pro",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "MBPP-Pro",
   "CodeEval-Pro MBPP"
  ],
  "category": "coding",
  "contamination": {
   "note": "Base problems come from public MBPP (2021). Companion problems, tests and solutions were generated with DeepSeek-V2.5 and released 31 December 2024 on Hugging Face. Shorter exposure than MBPP, but no longer new in 2026.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT",
   "modalities": [
    "text",
    "code"
   ],
   "public_test_set": true,
   "size": 378,
   "size_note": "Hugging Face CodeEval-Pro/mbpp-pro, default config, split named train, 378 rows (Hub API and datasets-server, accessed 2026-09-08). Fields: id, raw_problem, new_problem, raw_solution, new_solution, test_code. That 378 matches EvalPlus MBPP+'s current test size, not original MBPP's 974. The paper describes generation from original MBPP problems; the released file is the 378-row set.\n",
   "splits": "single 378-row split published as train; no separate official test split on the Hub",
   "url": "https://huggingface.co/datasets/CodeEval-Pro/mbpp-pro"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-006 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-006"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "mbpp_pro",
   "other": "Reference repo task types include mbpp_pro, mbpp_pro_cot, mbpp_pro_1shot. OpenCompass abbr mbpp_pro uses MBPPProDataset/MBPPProEvaluator against https://opencompass-multiple-evaluator.hf.space, not local execution."
  },
  "id": "mbpp_pro",
  "last_updated": "",
  "leaderboard_url": "https://answers111.github.io/evalpro.github.io/leaderboard.html",
  "lineage": {
   "family": "",
   "predecessor": "mbpp",
   "successors": [],
   "variants": [
    "humaneval_pro"
   ]
  },
  "measures": "MBPP Pro is the MBPP half of HumanEval Pro and MBPP Pro (Yu et al., 2024). Each item gives a base MBPP-style problem and a related, harder problem in one prompt. The model must emit working Python for both, and the second solution is meant to call the first rather than reimplement it. The paper calls this self-invoking code generation: composition, not isolated snippets. Hugging Face CodeEval-Pro/mbpp-pro holds 378 such pairs.\n",
  "metric": {
   "baseline_note": "Report the self-invoking pass@1 together with the matched base-problem score. Canonical solutions reach 100% pass@1 after three human-review rounds (paper Table 1: MBPP Pro 84.7 then 99.7 then 100.0). No random or human-solver baseline is given for models.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "pass@1",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "MBPP Pro",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2412.21199",
   "title": "HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation",
   "url": "https://arxiv.org/abs/2412.21199",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Zhaojian Yu",
    "Yilun Zhao",
    "Arman Cohan",
    "Xiao-Ping Zhang"
   ],
   "org": "Tsinghua University; Yale University",
   "url": "https://github.com/CodeEval-Pro/CodeEval-Pro"
  },
  "released": "2024-12",
  "repo_url": "https://github.com/CodeEval-Pro/CodeEval-Pro",
  "saturation": {
   "as_of": "2024-12",
   "note": "CodeEval-Pro leaderboard data.csv (fetched 2026-09-08, rows not individually dated) ranks greedy 0-shot MBPP Pro pass@1 with DeepseekCoder-V2-Instruct at 71.4, Deepseek-V2.5 at 71.2, GPT-4o at 70.9. DeepSeek-R1 is 68.8 here and 79.2 on HumanEval Pro, so the two Pro scores are not the same ranking. Spread down into the 30s\u201340s for smaller models is wide enough to call this open.\n",
   "status": "open",
   "top_score": 71.4
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HumanEval Pro and MBPP Pro (arXiv 2412.21199)",
    "url": "https://arxiv.org/abs/2412.21199"
   },
   {
    "accessed": "2026-09-08",
    "title": "CodeEval-Pro repository (released 2024-12-31 per README news)",
    "url": "https://github.com/CodeEval-Pro/CodeEval-Pro"
   },
   {
    "accessed": "2026-09-08",
    "title": "CodeEval-Pro/mbpp-pro Hub API (378 rows, MIT)",
    "url": "https://huggingface.co/api/datasets/CodeEval-Pro/mbpp-pro"
   },
   {
    "accessed": "2026-09-08",
    "title": "CodeEval-Pro leaderboard data.csv",
    "url": "https://answers111.github.io/evalpro.github.io/data.csv"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass mbpp_pro config (hosted evaluator)",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/mbpp_pro/mbpp_pro_gen_3dc067.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass mbpp_pro README (sample OC vs CodeEval-pro scores)",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/mbpp_pro/README.md"
   }
  ],
  "status": "active",
  "subcategory": "self-invoking Python generation built on MBPP problems",
  "summary": "CodeEval-Pro's harder MBPP: each base problem is paired with a second task the model must solve by calling its own solution to the first.",
  "tags": [
   "code-generation",
   "python",
   "self-invoking",
   "pass-at-k",
   "reasoning"
  ],
  "task_format": "Zero-shot (default) generation of two Python solutions in one response, from raw_problem and new_problem fields; graded by executing both against author-reviewed tests (pass@1). The repo also defines mbpp_pro_cot and mbpp_pro_1shot variants.\n"
 }
}