Inverse IFEval tests whether a model can override trained habits -- always answering, always correct, always-commented code -- to comply with instructions that deliberately conflict with them.
unassessed
| Category | instruction-following |
|---|---|
| Subcategory | counter-conventional instruction following: overriding SFT-trained habits to comply with instructions that conflict with them |
| Page status | active |
| Metric | binary pass rate under LLM-as-judge grading (0 or 1 per response, scored against the item's own bundled reference criteria), reported overall and per instruction type and language |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1012 |
| Publisher | Hosted under the m-a-p (Multimodal Art Projection) organisation on Hugging Face; individual authors' institutional affiliations were not confirmed from a source read for this page |
Inverse IFEval tests a model's ability to override the standardised habits instilled by supervised fine-tuning and comply with an instruction that deliberately conflicts with them, rather than testing whether it follows ordinary, cooperative instructions the way IFEval (in this repository) does. Its 1,012 prompts span eight challenge types: Question Correction (recognise that a multiple-choice question has no correct option, instead of picking the closest wrong one), Intentional Textual Flaws, Code without Comments, Counter-Conventional Formatting, Deliberately Incorrect Answers, Instructional Induction, Mid-turn Instruction Modification, and Counterfactual Answering. One released example asks a model to solve a rigged word problem whose four multiple-choice options are all numerically wrong; earning credit requires stating that none of the options is correct rather than defaulting to picking one, the trained habit the item is built to surface.
Single-turn free-text generation for most items; Mid-turn Instruction Modification introduces a change to the instruction partway through a multi-part prompt instead. Every item carries its own reference criteria describing exactly what a compliant response must and must not contain, and its own bundled LLM-judge system prompt and prompt template, rather than one fixed judge prompt applied across the whole dataset. Items were constructed through a human-in-the-loop pipeline across 23 domains, in matched Chinese and English versions.
No model card in ModelSpec reports this benchmark yet.