IFEvalCode scores code models on two independent checks per problem: functional correctness, and whether the code also obeys an explicit style or structural instruction, across eight languages.
unassessed
| Category | instruction-following |
|---|---|
| Subcategory | instruction-following applied to code generation: functional correctness and constraint adherence scored as two independent axes |
| Page status | active |
| Metric | correctness accuracy, instruction-adherence accuracy, and overall accuracy (share of samples passing both checks at once), reported overall and per language |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 810 |
| Publisher | Hosted under the Multilingual-Multimodal-NLP organisation on Hugging Face; individual authors' institutional affiliations were not confirmed from a source read for this page |
IFEvalCode gives a model a natural-language coding prompt that bundles an ordinary programming task with an explicit, checkable constraint on the code itself -- a naming convention, a required language construct such as a list comprehension or ternary operator, a formatting rule, or a ban on comments. Every one of the 810 problems is offered in both Chinese and English, and spans eight languages: Python, Java, JavaScript, TypeScript, C++, C#, PHP and Shell. The benchmark's stated point is that code-generation evaluations that check only correctness miss a second, largely independent failure mode: a model can write working code that ignores the instructions layered on top of the underlying task, the same gap IFEval (in this repository) measures for natural-language responses.
Single-turn code generation: given a prompt combining a coding task with an instruction constraint, the model produces a solution for a named entry-point function. Each problem ships its own executable `check_correctness` function (functional test cases) and `check_instruction` function (a language-aware static or regex check of the submitted code, for example scanning variable names for snake_case or checking that no list comprehension appears in the source). Execution happens inside a per-language Docker sandbox.
No model card in ModelSpec reports this benchmark yet.