{
 "body": "## What it measures\n\nEvoEval tests whether language models can synthesize working programs from natural-language specifications when the problem is less familiar than a standard benchmark item. The project evolves existing coding problems into targeted domains called Difficult, Creative, Subtle, Combine, and Tool Use.\n\nThe benchmark exercises Python-style program synthesis and the ability to compose concepts or use auxiliary functions. It was designed to expose overfitting to popular, publicly discussed coding tasks.\n\n## How it is scored\n\nSolutions are evaluated by running generated code against the benchmark\u2019s tests. The primary measure is the share of problems solved. The project reports a mean 39.6% performance drop from standard benchmarks across its study, with model-specific drops from 20.0% to 47.7%; those are study comparisons, not a universal baseline. Tool-use variants require the model to call provided helpers correctly.\n\n## Dataset and licence\n\nThe project page describes evolved problems derived from existing coding benchmarks. It does not state one consolidated item count or licence on the page consulted here. The benchmark and tests are publicly distributed through the project, so answer leakage remains possible. Exact split counts and licence should be checked in the reference repository before reuse.\n\n## Who publishes it\n\nEvoEval is maintained by the project authors behind the EvalPlus ecosystem. The official project page is the primary source consulted and reports a study of 51 LLMs. No current independent leaderboard is established there.\n\n## Lineage\n\nEvoEval explicitly evolves existing benchmarks such as HumanEval. Difficult, Creative, Subtle, Combine, and Tool Use are its named variants. The project does not identify a successor benchmark on the official page.\n\n## Saturation and contamination\n\nThe reported drops and changed model rankings show that EvoEval remains open relative to standard coding tests. Its motivation is leakage: older benchmark examples and solutions are widely available. Because EvoEval itself is public, later training exposure is possible; contamination is medium unless a private or newly refreshed split is used.\n\n## How to run it\n\nUse the official project data and execution harness. Report the suite variant, programming language, prompting format, sampling count, timeout, and test runner. Compare with the corresponding source benchmark under the same model settings; otherwise the performance drop can mix task novelty with protocol changes.\n\n## Reading the numbers\n\nA high pass rate indicates that generated code passes the supplied tests for the selected evolved tasks. It does not prove general software engineering ability or resistance to hidden tests. Compare across EvoEval variants because they stress different failure modes. Inspect tool-use traces and failures when evaluating agentic coding systems.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "evo_eval",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "category": "coding",
  "contamination": {
   "note": "EvoEval was designed to reduce leakage from older benchmark problems, but evolved items remain publicly available.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "modalities": [
    "code"
   ],
   "public_test_set": true,
   "url": "https://evo-eval.github.io/"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna, luna-stream-a-004 (Codex coordinated)",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "other": "EvoEval reference evaluation scripts."
  },
  "id": "evo_eval",
  "leaderboard_url": "https://evo-eval.github.io/",
  "measures": "EvoEval measures program synthesis beyond familiar benchmark items by transforming existing coding problems into targeted variants. Its suites test difficult reasoning, creative solutions, subtle edge cases, composition, and use of provided tools.",
  "metric": {
   "direction": "higher_is_better",
   "max_score": 100,
   "name": "pass rate",
   "unit": "percent"
  },
  "name": "EvoEval",
  "page_kind": "benchmark",
  "paper": {},
  "publisher": {
   "org": "EvoEval authors",
   "url": "https://evo-eval.github.io/"
  },
  "released": "2023",
  "repo_url": "https://github.com/evalplus/evoeval",
  "saturation": {
   "note": "The project reports substantial performance drops relative to standard coding benchmarks.",
   "status": "open"
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Official EvoEval project page",
    "url": "https://evo-eval.github.io/"
   }
  ],
  "summary": "EvoEval evaluates code generation on evolved HumanEval-style problems across difficult, creative, subtle, combined, and tool-use domains.",
  "tags": [
   "code-generation",
   "leakage",
   "program-synthesis"
  ],
  "task_format": "Code generation from natural-language problem statements, scored by execution tests."
 }
}