{
 "body": "\n## What it measures\n\nHumanEval Pro pairs each of HumanEval's 164 problems with a second, harder problem that is designed\nto be solved by calling the first problem's own solution. A model is given both problems in one\nprompt and must produce working Python for each, with the second implementation expected to invoke\nthe first rather than reimplement its logic from scratch. The paper calls this \"self-invoking code\ngeneration\": it is meant to test whether a model that can already write a correct function can also\ncompose that function into something more complex, a step closer to how real code gets built than one\nisolated function at a time.\n\n## How it is scored\n\nBoth the base solution and the self-invoking solution are executed against test suites and checked\nfor a pass, reported as pass@1 under greedy decoding. Results are typically reported two ways for the\nsame model: its score on the base (HumanEval-style) problem alone, and its score on the combined\nself-invoking task, so a single \"HumanEval Pro\" figure without its paired base score is only half the\ncomparison the benchmark is built to support. The authors built each self-invoking problem, its\ncandidate solution and its test inputs with DeepSeek-V2.5, executed the candidate solutions to obtain\nground-truth outputs, and then had human experts iteratively review and correct the test cases and\ncanonical solutions, reporting a 100% pass@1 for their own reference solutions under this process.\n\n## Dataset and licence\n\nHumanEval Pro contains 164 problems, one self-invoking pair per original HumanEval problem, confirmed\nby row count (164) on the CodeEval-Pro/humaneval-pro dataset through the Hugging Face\ndatasets-server. The dataset is released under the MIT licence on Hugging Face\n(huggingface.co/CodeEval-Pro), alongside a sibling MBPP Pro dataset built the same way from MBPP and a\nsmaller BigCodeBench-Lite Pro (57 problems) built from BigCodeBench. All prompts are in English; the\ntarget language is Python.\n\n## Who publishes it\n\nHumanEval Pro was introduced by Zhaojian Yu and Xiao-Ping Zhang (Tsinghua University) with Yilun Zhao\nand Arman Cohan (Yale University) in \"HumanEval Pro and MBPP Pro: Evaluating Large Language Models on\nSelf-invoking Code Generation,\" posted to arXiv in December 2024 and later accepted to ACL 2025\nFindings. The authors maintain the reference implementation and leaderboard as \"CodeEval-Pro\"\n(github.com/CodeEval-Pro and a matching Hugging Face dataset organisation).\n\n## Lineage\n\nHumanEval Pro's predecessor is HumanEval (this repository's humaneval page): every base problem is\none of HumanEval's original 164, and the paper explicitly frames the benchmark as a harder successor\nbuilt because \"most LLMs excel in traditional code generation benchmarks like HumanEval and MBPP, but\ntheir performance declines on self-invoking tasks.\" The same paper introduces two sibling benchmarks\nbuilt the same way -- MBPP Pro (from MBPP) and BigCodeBench-Lite Pro (from BigCodeBench) -- which are\nnot descendants of HumanEval Pro itself and do not have pages in this repository. OpenCompass\nseparately packages this benchmark as its own `humaneval_pro` config, evaluated through a hosted\nevaluator rather than the authors' own harness.\n\n## Saturation and contamination\n\nOn the CodeEval-Pro leaderboard (accessed 2026-09-08, entries not individually dated), the top\nself-invoking pass@1 is 79.2 (DeepSeek-R1), with scores spreading down to the 30s for smaller or older\nbase models -- a wide, real spread that separates models clearly, so this is graded \"open.\" That is\nprecisely the benchmark's stated purpose: the paper reports every evaluated model scoring lower on the\nself-invoking task than on the matched base problem, for example o1-mini at 96.2% zero-shot HumanEval\nversus 76.2% HumanEval Pro, a gap plain HumanEval cannot show because it does not test composition.\nContamination risk is graded medium: the base half of every problem is drawn from HumanEval, public\nand exposed since July 2021, but the self-invoking companion problems, their solutions and their test\ncases were generated in late 2024 and have only been public since the dataset's December 2024 release,\na much shorter exposure window than HumanEval's own.\n\n## How to run it\n\nThe authors' reference implementation (github.com/CodeEval-Pro/CodeEval-Pro) supports `humaneval_pro`\nas one of several task types (alongside `mbpp_pro` and chain-of-thought or 1-shot variants),\ngenerating with vllm for local models or a direct API call for hosted ones. OpenCompass ships a\n`humaneval_pro` config that instead scores completions through a separately hosted evaluator service,\nso OpenCompass results and the authors' own scripts are not guaranteed to share identical grading\ninfrastructure, even though both report pass@1.\n\n## Reading the numbers\n\nA high HumanEval Pro score is a stronger claim than a high HumanEval score: it means the model can\nboth solve a self-contained function and correctly call that function's own solution while solving a\nsecond, related problem, not just produce isolated snippets. The gap between a model's HumanEval and\nHumanEval Pro scores is itself a useful number -- a small gap suggests the model's coding ability\nactually composes, a large one suggests its HumanEval score overstates practical coding skill. As with\nany benchmark released in December 2024, treat scores reported from before that date as impossible,\nand stay alert to newer models trained on data that could include the now-public reference solutions.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "humaneval_pro",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "coding",
  "contamination": {
   "note": "The base half of every problem is drawn from HumanEval, public and exposed since July 2021, but the self-invoking companion problems, their solutions and their test cases were generated in late 2024 and have only been public since the dataset's December 2024 release -- a much shorter exposure window than HumanEval's own, though no longer a brand-new one by 2026.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "license": "MIT",
   "modalities": [
    "text",
    "code"
   ],
   "public_test_set": true,
   "size": 164,
   "size_note": "164 problems, one self-invoking pair per original HumanEval problem, confirmed by row count (164) on the CodeEval-Pro/humaneval-pro dataset through the Hugging Face datasets-server.\n",
   "splits": "single split, 164 rows; no train or validation split",
   "url": "https://huggingface.co/datasets/CodeEval-Pro/humaneval-pro"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 6, slice A",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "humaneval_pro",
   "other": "Reference implementation: github.com/CodeEval-Pro/CodeEval-Pro, which supports humaneval_pro alongside mbpp_pro and chain-of-thought or 1-shot variants, generating with vllm for local models or a direct API call for hosted ones. OpenCompass's humaneval_pro config instead scores completions through a separately hosted evaluator service rather than executing locally."
  },
  "id": "humaneval_pro",
  "last_updated": "",
  "leaderboard_url": "https://answers111.github.io/evalpro.github.io/leaderboard.html",
  "lineage": {
   "family": "humaneval",
   "predecessor": "humaneval",
   "successors": [],
   "variants": []
  },
  "measures": "HumanEval Pro pairs each of HumanEval's 164 problems with a second, harder problem designed to be solved by calling the first problem's own solution. A model is given both problems in one prompt and must produce working Python for each, with the second implementation expected to invoke the first rather than reimplement its logic from scratch. The paper calls this \"self-invoking code generation\": it tests whether a model that can already write a correct function can also compose that function into something more complex, closer to how real code gets built than one isolated function at a time.\n",
  "metric": {
   "baseline_note": "Typically reported alongside the model's score on the matched base (HumanEval-style) problem, so a single \"HumanEval Pro\" figure without its paired base score is only half the comparison the benchmark is built to support. The original paper's own results show every evaluated model scoring lower on the self-invoking pass@1 than on the corresponding base or plain-HumanEval pass@1, for example o1-mini at 96.2% zero-shot HumanEval versus 76.2% HumanEval Pro. No random-guess or human baseline is established.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "pass@1",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "HumanEval Pro",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2412.21199",
   "title": "HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation",
   "url": "https://arxiv.org/abs/2412.21199",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Zhaojian Yu",
    "Yilun Zhao",
    "Arman Cohan",
    "Xiao-Ping Zhang"
   ],
   "org": "Tsinghua University; Yale University",
   "url": "https://github.com/CodeEval-Pro/CodeEval-Pro"
  },
  "released": "2024-12",
  "repo_url": "https://github.com/CodeEval-Pro/CodeEval-Pro",
  "saturation": {
   "as_of": "",
   "note": "On the CodeEval-Pro leaderboard (accessed 2026-09-08, entries not individually dated), the top self-invoking pass@1 is 79.2 (DeepSeek-R1), with scores spreading down to the 30s for smaller or older base models -- a wide, real spread that separates models clearly, hence \"open.\" That is precisely the benchmark's stated purpose: the paper reports every evaluated model scoring lower on the self-invoking task than on its matched base problem, a gap plain HumanEval cannot show because it does not test composition.\n",
   "status": "open",
   "top_score": 79.2
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation",
    "url": "https://arxiv.org/abs/2412.21199"
   },
   {
    "accessed": "2026-09-08",
    "title": "CodeEval-Pro/CodeEval-Pro repository",
    "url": "https://github.com/CodeEval-Pro/CodeEval-Pro"
   },
   {
    "accessed": "2026-09-08",
    "title": "CodeEval-Pro/humaneval-pro dataset card",
    "url": "https://huggingface.co/datasets/CodeEval-Pro/humaneval-pro"
   },
   {
    "accessed": "2026-09-08",
    "title": "CodeEval-Pro Leaderboard",
    "url": "https://answers111.github.io/evalpro.github.io/leaderboard.html"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass humaneval_pro config",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/humaneval_pro/humaneval_pro_gen_3dc067.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass humaneval_pro README (sample results)",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/humaneval_pro/README.md"
   }
  ],
  "status": "active",
  "subcategory": "self-invoking code generation",
  "summary": "A harder successor to HumanEval that pairs each of its 164 problems with a second, more complex problem the model must solve by correctly invoking its own solution to the first.",
  "tags": [
   "code-generation",
   "python",
   "self-invoking",
   "pass-at-k",
   "functional-correctness",
   "reasoning"
  ],
  "task_format": "Given a base problem (one of HumanEval's 164) and a related, harder problem in the same prompt, generate Python solutions to both in one response, where the second solution is meant to call the first; graded by executing both against test suites the authors built and manually reviewed (pass@1).\n"
 }
}