{
 "body": "\n## What it measures\n\nHumanEval-Infilling tests fill-in-the-middle completion: given the code both before and after a gap,\ncan a model produce what belongs in the gap, rather than only continuing text left to right. Each\ntask takes one of HumanEval's 164 canonical solutions and removes a piece of it -- a single line, a\nrun of consecutive lines, or a randomly chosen span of characters -- leaving the model the docstring,\nthe code before the gap, and the code after it. This is a different skill from HumanEval's own\ncode-generation task: a model that writes a correct function from scratch is not automatically good\nat inserting a missing piece into someone else's code with both sides already fixed, which is closer\nto how code gets edited in practice than to writing it from nothing.\n\n## How it is scored\n\nA completion is checked two ways: whether the reassembled function (prefix plus generated span plus\nsuffix) passes HumanEval's original unit tests, and whether the generated span exactly matches the\ntext that was removed. The lm-evaluation-harness implementation reports pass@1 and aggregates the\nfour subsets (single-line, multi-line, random-span, random-span-light) with an unweighted mean,\ndespite the subsets ranging from 164 to 5,815 tasks, so a single reported score can be dominated by\nwhichever subset happens to score best or worst rather than reflecting task volume.\n\n## Dataset and licence\n\nThere is no single dataset size: HumanEval-Infilling packages four differently sized task subsets\nbuilt by masking HumanEval's 164 canonical solutions. Per the lm-evaluation-harness task README,\nsingle-line infilling has 1,033 tasks (one per non-blank line across all 164 solutions), multi-line\nhas 5,815 (one per contiguous span of non-blank lines), random-span has 1,640, and random-span-light\n-- a smaller, faster-to-run version -- has 164, one per problem. The reference repository,\nopenai/human-eval-infilling, is released under the MIT licence. Prompts are in English; the only\nprogramming language covered is Python.\n\n## Who publishes it\n\nThe bundled benchmark and its reference harness were released by OpenAI's Mohammad Bavarian, Heewoo\nJun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek and Mark Chen in \"Efficient\nTraining of Language Models to Fill in the Middle,\" posted to arXiv in July 2022. Two of its four\nsubsets are not new to that paper: the paper's own text credits the single-line and multi-line\ninfilling tasks to Daniel Fried and co-authors' InCoder paper (\"Fried et al. 2022\"), which built them\nby masking non-blank lines out of HumanEval's canonical solutions several months earlier, confirmed\ndirectly in InCoder's own paper. OpenAI's own contribution was the two random-span variants, plus\npackaging all four as one released benchmark and harness. OpenAI maintains the\nopenai/human-eval-infilling repository.\n\n## Lineage\n\nHumanEval-Infilling's predecessor is HumanEval (this repository's humaneval page): every task is\nbuilt by masking spans out of HumanEval's own canonical solutions, so its 164-problem base carries\nthe same age and exposure as the parent. Its own lineage is unusually split: half of its four subsets\n(single-line, multi-line) trace to Daniel Fried et al.'s InCoder paper (arXiv 2204.05999, April\n2022), which does not have its own page in this repository, while the other half (random-span,\nrandom-span-light) and the bundled repository that packages all four originate in OpenAI's later FIM\npaper (arXiv 2207.14255, July 2022). A report that simply cites \"the InCoder benchmark\" or \"the FIM\nbenchmark\" is describing only part of what the id humaneval_infilling actually runs.\n\n## Saturation and contamination\n\nNo current, actively maintained leaderboard for HumanEval-Infilling could be found, so saturation is\ngraded \"unknown\" rather than inferred from either paper's now-dated model comparisons. Contamination\nrisk is high, and arguably sharper than for an ordinary HumanEval score: every task's expected answer\nis a verbatim substring of a HumanEval canonical solution that has been sitting in a public GitHub\nrepository since July 2021, so a model that has simply memorised HumanEval's solutions can reconstruct\nmany infilling spans without any real fill-in-the-middle ability.\n\n## How to run it\n\nlm-evaluation-harness implements all four subsets under the task group `humaneval_infilling`\n(individually `humaneval_single_line_infilling`, `humaneval_multi_line_infilling`,\n`humaneval_random_span_infilling`, `humaneval_random_span_infilling_light`), built on the `code_eval`\nmetric from Hugging Face's `evaluate` package, which -- like the original HumanEval harness --\nexecutes untrusted, model-generated code and requires deliberately enabling that behaviour. The\noriginal reference implementation, openai/human-eval-infilling, provides the same four benchmarks\ndirectly.\n\n## Reading the numbers\n\nA high HumanEval-Infilling score shows a model can use context on both sides of a gap, useful signal\nfor editor-style code completion rather than generate-from-scratch coding. Because every answer is\ndrawn from HumanEval's long-public solutions, a high score is even less informative about genuine\ninfilling ability than a high HumanEval score is about genuine synthesis ability -- check whether a\nmodel was evaluated on this benchmark specifically for fill-in-the-middle capability (typically base,\ncompletion-focused code models) rather than assuming a chat-tuned model was tested on it at all.\nBecause the four subsets are unevenly sized and averaged without weighting, look at the per-subset\nbreakdown, not just one blended number, before comparing two models.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "humaneval_infilling",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "coding",
  "contamination": {
   "note": "Contamination risk here is arguably sharper than for an ordinary HumanEval score: every task's expected answer is a verbatim substring of a HumanEval canonical solution that has been sitting in a public GitHub repository since July 2021, so a model that has simply memorised HumanEval's solutions can reconstruct many infilling spans without any real fill-in-the-middle ability.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "license": "MIT",
   "modalities": [
    "text",
    "code"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "No single size: four differently sized subsets are built by masking spans out of HumanEval's 164 canonical solutions. Per the lm-evaluation-harness task README, single-line infilling has 1,033 tasks (one per non-blank line across all 164 solutions), multi-line has 5,815 (one per contiguous span of non-blank lines), random-span has 1,640, and random-span-light -- a smaller, faster version -- has 164, one per problem.\n",
   "splits": "four task subsets (single-line, multi-line, random-span, random-span-light), each derived from HumanEval's 164-problem test set; no train split",
   "url": "https://github.com/openai/human-eval-infilling"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 6, slice A",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "humaneval_infilling",
   "opencompass": "",
   "other": "lm-evaluation-harness implements the group `humaneval_infilling` over four child tasks (humaneval_single_line_infilling, humaneval_multi_line_infilling, humaneval_random_span_infilling, humaneval_random_span_infilling_light), built on Hugging Face's `evaluate` code_eval metric. The original reference implementation is openai/human-eval-infilling."
  },
  "id": "humaneval_infilling",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "humaneval",
   "predecessor": "humaneval",
   "successors": [],
   "variants": []
  },
  "measures": "HumanEval-Infilling tests a model's ability to fill in a missing piece of code given the text on both sides of the gap, rather than only continuing left to right. Each task takes one of HumanEval's 164 canonical solutions and removes a span of it -- a single line, several consecutive lines, or a randomly chosen character span -- so the model must produce the missing piece from the docstring, the code before the gap, and the code after it. This exercises a different skill from HumanEval's own generate-from-scratch task, closer to how code gets edited than to how it gets written new.\n",
  "metric": {
   "baseline_note": "lm-evaluation-harness reports pass@1 per subset and aggregates the four subsets (single-line, multi-line, random-span, random-span-light) with an unweighted mean, despite the subsets ranging from 164 to 5,815 tasks, so one blended score can be dominated by whichever subset happens to score best or worst. The InCoder and FIM papers also report exact match as a secondary metric. No random-guess or human baseline is established.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "pass@1",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "HumanEval-Infilling",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2207.14255",
   "title": "Efficient Training of Language Models to Fill in the Middle",
   "url": "https://arxiv.org/abs/2207.14255",
   "year": 2022
  },
  "publisher": {
   "authors": [
    "Mohammad Bavarian",
    "Heewoo Jun",
    "Nikolas Tezak",
    "John Schulman",
    "Christine McLeavey",
    "Jerry Tworek",
    "Mark Chen"
   ],
   "org": "OpenAI",
   "url": "https://github.com/openai/human-eval-infilling"
  },
  "released": "2022-07",
  "repo_url": "https://github.com/openai/human-eval-infilling",
  "saturation": {
   "as_of": "",
   "note": "No current, actively maintained leaderboard for HumanEval-Infilling could be found, so saturation is graded \"unknown\" rather than inferred from either source paper's now-dated model comparisons.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Efficient Training of Language Models to Fill in the Middle",
    "url": "https://arxiv.org/abs/2207.14255"
   },
   {
    "accessed": "2026-09-08",
    "title": "InCoder: A Generative Model for Code Infilling and Synthesis",
    "url": "https://arxiv.org/abs/2204.05999"
   },
   {
    "accessed": "2026-09-08",
    "title": "Efficient Training of Language Models to Fill in the Middle (full text, for the Fried et al. 2022 attribution of the single/multi-line tasks)",
    "url": "https://ar5iv.labs.arxiv.org/html/2207.14255"
   },
   {
    "accessed": "2026-09-08",
    "title": "openai/human-eval-infilling repository",
    "url": "https://github.com/openai/human-eval-infilling"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness: humaneval_infilling task README",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/humaneval_infilling/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness: humaneval_infilling group config",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/humaneval_infilling/humaneval_infilling.yaml"
   }
  ],
  "status": "active",
  "subcategory": "code infilling / fill-in-the-middle",
  "summary": "Four fill-in-the-middle tasks built by masking spans of HumanEval's solutions; two (single/multi-line) come from the InCoder paper, two (random-span) were added by OpenAI's FIM paper.",
  "tags": [
   "code-generation",
   "python",
   "fill-in-the-middle",
   "infilling",
   "pass-at-k",
   "functional-correctness"
  ],
  "task_format": "Given a function's prefix and suffix with a span of the canonical solution removed (a single line, several consecutive lines, or a random character span), generate the missing span; the reassembled function is graded by executing it against HumanEval's original unit tests (pass@k), alongside an exact-match check against the removed text.\n"
 }
}