EvalPlus's stricter HumanEval, testing the same 164 Python problems against roughly 80x more unit tests so incorrect completions that pass the original suite get caught.
unassessed
| Category | coding |
|---|---|
| Subcategory | function-level code generation |
| Page status | active |
| Metric | pass@1 |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 164 |
| Dataset licence | Apache-2.0 |
| Publisher | University of Illinois Urbana-Champaign; Nanjing University |
HumanEval+ asks a model to complete the same 164 hand-written Python functions as HumanEval, from the same signatures and docstrings. What differs is not the prompt but the grading: EvalPlus augments each problem's original, sparse test suite with a much larger set of automatically generated edge-case tests, built with both LLM-based and mutation-based test-input generators. A completion that looks correct under HumanEval's original handful of tests can still fail one of the new ones, so HumanEval+ measures the same coding skill as HumanEval but checks it far more rigorously, and the two scores are routinely conflated under the single name "HumanEval."
Complete a Python function body from its signature, docstring and any starter code, identical to HumanEval; graded by executing the completion against EvalPlus's expanded test suite (pass@k).
No model card in ModelSpec reports this benchmark yet.