Four fill-in-the-middle tasks built by masking spans of HumanEval's solutions; two (single/multi-line) come from the InCoder paper, two (random-span) were added by OpenAI's FIM paper.
unassessed
| Category | coding |
|---|---|
| Subcategory | code infilling / fill-in-the-middle |
| Page status | active |
| Metric | pass@1 |
| Direction | higher_is_better |
| Unit | % |
| Dataset licence | MIT |
| Publisher | OpenAI |
HumanEval-Infilling tests a model's ability to fill in a missing piece of code given the text on both sides of the gap, rather than only continuing left to right. Each task takes one of HumanEval's 164 canonical solutions and removes a span of it -- a single line, several consecutive lines, or a randomly chosen character span -- so the model must produce the missing piece from the docstring, the code before the gap, and the code after it. This exercises a different skill from HumanEval's own generate-from-scratch task, closer to how code gets edited than to how it gets written new.
Given a function's prefix and suffix with a span of the canonical solution removed (a single line, several consecutive lines, or a random character span), generate the missing span; the reassembled function is graded by executing it against HumanEval's original unit tests (pass@k), alongside an exact-match check against the removed text.
No model card in ModelSpec reports this benchmark yet.