{
 "body": "\n## What it measures\n\nHumanEval+ gives a model the exact same 164 Python problems as HumanEval: a function signature, an\nEnglish docstring, and sometimes example input/output pairs, with the model completing the body.\nNothing about the prompt changes. What changes is how the completion gets checked afterward, which is\nwhy HumanEval+ is routinely and incorrectly reported as if it were the original HumanEval number.\n\n## How it is scored\n\nCorrectness is still checked by execution, but against a much larger test suite. EvalPlus augments\neach of HumanEval's original hidden tests with test inputs generated automatically, using both\nLLM-based and mutation-based generation strategies aimed at edge cases the hand-written originals\nmiss. The project's own documentation puts the expansion at roughly 80x more tests per problem than\nthe original HumanEval suite. The metric is still pass@1 (or pass@10/pass@100 from repeated\nsampling), computed the same way as HumanEval's; what differs is the pass/fail line each completion\nhas to clear. The EvalPlus paper reports this drops measured pass@k by up to 19.3 to 28.9 percentage\npoints relative to the same completions graded on the original HumanEval tests, and can change which\nmodel looks best: WizardCoder-CodeLlama and Phind-CodeLlama both overtook ChatGPT on HumanEval+\ndespite neither beating it on HumanEval itself.\n\n## Dataset and licence\n\nThe problem set is unchanged from HumanEval -- 164 hand-written Python problems, single test split,\nno train or validation data -- confirmed against the evalplus/humanevalplus dataset card on Hugging\nFace. EvalPlus publishes its enhanced tests and prompts under the Apache-2.0 licence. All text is in\nEnglish; the only programming language covered is Python, matching the parent benchmark.\n\n## Who publishes it\n\nHumanEval+ was introduced by Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang and Lingming Zhang\n(University of Illinois Urbana-Champaign, with Yuyao Wang at Nanjing University) in \"Is Your Code\nGenerated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code\nGeneration,\" posted to arXiv in May 2023 and later published at NeurIPS 2023. The EvalPlus team\nmaintains the reference implementation, dataset and leaderboard at evalplus.github.io and\ngithub.com/evalplus/evalplus.\n\n## Lineage\n\nHumanEval+'s predecessor is HumanEval (this repository's humaneval page): same problems, same\nprompts, stricter grading only. EvalPlus applies the identical augmented-testing approach to MBPP as\nMBPP+, which is outside this benchmark's scope. A further, harder successor exists in this\nrepository's coding lineage: humaneval_pro pairs each HumanEval problem with a second, self-invoking\nproblem that must call the first one's solution, and shows the same strong models losing further\nground versus their HumanEval+ score. HumanEval+ itself has no further \"plus-plus\" revision;\nEvalPlus's own versioning instead periodically corrects individual ground-truth solutions and test\ncontracts (its changelog documents several such fixes) without renaming the benchmark.\n\n## Saturation and contamination\n\nOn the EvalPlus leaderboard (accessed 2026-09-08), the top pass@1 shown on the HumanEval \"EvalPlus\nTests\" view is 89, held jointly by two OpenAI o1-family models, with a wide spread down into the 30s\nfor smaller or older models. No entry on that leaderboard is dated later than November 2024, so it\nlikely understates where 2025-2026 frontier models would land and should be read as dated.\nContamination risk is high: HumanEval+ reuses HumanEval's own prompts and canonical solutions, public\nsince July 2021, so a model that has memorised the original benchmark starts every HumanEval+ problem\nwith the same advantage. The larger test suite does not reduce this exposure; it is designed instead\nto make superficial memorisation or narrow overfitting insufficient for a passing score, which is why\nscores drop under it even for models evaluated well after the original, smaller suite was public.\n\n## How to run it\n\nThe reference implementation is the evalplus PyPI package (github.com/evalplus/evalplus), which can\nscore a set of HumanEval completions against either the original tests or the expanded HumanEval+\ntests, which is the default. bigcode-evaluation-harness wraps the same dataset as task\n`humanevalplus`. OpenCompass's `humaneval_plus` config calls the evalplus package directly through\nits own `HumanEvalPlusEvaluator`. Because the same completions can be scored two different ways with\nthe same tool, always confirm whether a reported \"HumanEval\" number used the base or the plus tests\nbefore comparing it to another paper's.\n\n## Reading the numbers\n\nA high HumanEval+ pass@1 is a stronger signal of correct code than the same score on plain HumanEval,\nbecause it has survived roughly 80x more tests per problem, including edge cases the original suite\nmissed. A big gap between a model's HumanEval and HumanEval+ scores is itself informative: it usually\nmeans the model's HumanEval solutions are fragile rather than genuinely correct, and a leaderboard\nthat mixes the two views without labelling them can make two different measurements look like one. It\nstill shares HumanEval's other limits: short, self-contained Python functions only, prompts and\nreference solutions that have been public for years, and no test of working across files or tools.\nAlways check which test suite -- original or plus -- a reported \"HumanEval\" number actually used.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "humaneval_plus",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "HumanEval Plus"
  ],
  "category": "coding",
  "contamination": {
   "note": "HumanEval+ reuses HumanEval's own 164 prompts and canonical solutions verbatim, public since July 2021, so the same memorisation risk applies on the prompt side. The point of the larger test suite is not to reduce that exposure but to make superficial memorisation or overfitting less sufficient for a passing score: the EvalPlus paper reports its harder tests catch a substantial share of LLM-synthesized code that the original, smaller HumanEval suite let through.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "license": "Apache-2.0",
   "modalities": [
    "text",
    "code"
   ],
   "public_test_set": true,
   "size": 164,
   "size_note": "Same 164 problems as HumanEval, confirmed against the evalplus/humanevalplus dataset card on Hugging Face (single \"test\" split, 164 rows). What EvalPlus adds is not more problems but more tests per problem: the project's own documentation states HumanEval+ carries roughly 80x more test cases than the original HumanEval suite.\n",
   "splits": "single 164-problem test set, same structure as HumanEval; no train or validation split",
   "url": "https://huggingface.co/datasets/evalplus/humanevalplus"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 6, slice A",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "humaneval_plus",
   "other": "Reference implementation: the evalplus PyPI package / github.com/evalplus/evalplus, which grades a HumanEval submission against either the original tests (a base-only mode) or the expanded HumanEval+ tests (the default). bigcode-evaluation-harness wraps the same dataset as task humanevalplus. OpenCompass's humaneval_plus config calls the evalplus package directly through a HumanEvalPlusEvaluator."
  },
  "id": "humaneval_plus",
  "last_updated": "",
  "leaderboard_url": "https://evalplus.github.io/leaderboard.html",
  "lineage": {
   "family": "humaneval",
   "predecessor": "humaneval",
   "successors": [],
   "variants": []
  },
  "measures": "HumanEval+ asks a model to complete the same 164 hand-written Python functions as HumanEval, from the same signatures and docstrings. What differs is not the prompt but the grading: EvalPlus augments each problem's original, sparse test suite with a much larger set of automatically generated edge-case tests, built with both LLM-based and mutation-based test-input generators. A completion that looks correct under HumanEval's original handful of tests can still fail one of the new ones, so HumanEval+ measures the same coding skill as HumanEval but checks it far more rigorously, and the two scores are routinely conflated under the single name \"HumanEval.\"\n",
  "metric": {
   "baseline_note": "Same pass@1 definition as HumanEval (the fraction of problems solved by one sample, or estimated from repeated sampling with the unbiased pass@k estimator for pass@10/pass@100), computed against EvalPlus's expanded tests rather than the original ones. The EvalPlus paper reports this stricter grading reduces measured pass@k by up to 19.3-28.9 percentage points versus the same completions scored on original HumanEval, and that it can reorder model rankings: the paper reports WizardCoder-CodeLlama and Phind-CodeLlama both overtaking ChatGPT on HumanEval+ despite neither beating it on the original benchmark. No random-guess or human baseline is established.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "pass@1",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "HumanEval+",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2305.01210",
   "title": "Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation",
   "url": "https://arxiv.org/abs/2305.01210",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Jiawei Liu",
    "Chunqiu Steven Xia",
    "Yuyao Wang",
    "Lingming Zhang"
   ],
   "org": "University of Illinois Urbana-Champaign; Nanjing University",
   "url": "https://github.com/evalplus/evalplus"
  },
  "released": "2023-05",
  "repo_url": "https://github.com/evalplus/evalplus",
  "saturation": {
   "as_of": "",
   "note": "The EvalPlus leaderboard (accessed 2026-09-08), on its default HumanEval \"EvalPlus Tests\" view, showed a top pass@1 of 89, held jointly by two OpenAI o1-family models, with a wide spread down into the 30s for older or smaller models. No entry on that leaderboard is dated later than November 2024, so it does not reflect any model released in 2025 or 2026 and the 89 top score should be read as dated rather than current. A further, harder self-invoking successor built on the same problems (humaneval_pro) shows the same top models dropping into the 70s, which is why this is graded \"watch\" rather than \"open.\"\n",
   "status": "watch",
   "top_score": 89.0
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation",
    "url": "https://arxiv.org/abs/2305.01210"
   },
   {
    "accessed": "2026-09-08",
    "title": "evalplus/evalplus repository",
    "url": "https://github.com/evalplus/evalplus"
   },
   {
    "accessed": "2026-09-08",
    "title": "evalplus/humanevalplus dataset card",
    "url": "https://huggingface.co/datasets/evalplus/humanevalplus"
   },
   {
    "accessed": "2026-09-08",
    "title": "EvalPlus Leaderboard",
    "url": "https://evalplus.github.io/leaderboard.html"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass humaneval_plus config",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/humaneval_plus/humaneval_plus_gen_8e312c.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass HumanEvalPlusEvaluator source",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/datasets/humaneval.py"
   }
  ],
  "status": "active",
  "subcategory": "function-level code generation",
  "summary": "EvalPlus's stricter HumanEval, testing the same 164 Python problems against roughly 80x more unit tests so incorrect completions that pass the original suite get caught.",
  "tags": [
   "code-generation",
   "python",
   "pass-at-k",
   "functional-correctness",
   "unit-tests",
   "evalplus"
  ],
  "task_format": "Complete a Python function body from its signature, docstring and any starter code, identical to HumanEval; graded by executing the completion against EvalPlus's expanded test suite (pass@k)."
 }
}