{
 "body": "\n## What it measures\n\nInverse IFEval tests whether a model can override the standardised habits instilled by supervised\nfine-tuning -- always give an answer, always write correct code, always add comments, always follow a\nlearned formatting convention -- when an instruction explicitly asks it to do otherwise. That is close to\nthe opposite of what IFEval (in this repository) tests: IFEval checks compliance with ordinary,\ncooperative verifiable instructions, while Inverse IFEval checks compliance when the instruction runs\nagainst what training has taught the model to do by default. Its 1,012 prompts span eight challenge types,\nincluding Question Correction, Code without Comments, Deliberately Incorrect Answers and Counterfactual\nAnswering. One released item asks a model to solve a rigged word problem whose four multiple-choice options\nare all numerically wrong; the correct behaviour is to state that none of the options is right rather than\ndefault to picking the closest one, which is exactly the trained habit the item is designed to surface.\n\n## How it is scored\n\nMost items are single-turn free-text generation; Mid-turn Instruction Modification instead introduces a\nchange to the instruction partway through a multi-part prompt, testing whether the model keeps following\nthe update rather than the original framing. Every item carries its own reference criteria spelling out\nwhat a compliant response must and must not do, together with its own bundled LLM-judge system prompt and\nprompt template -- an unusual design choice, since most LLM-judged benchmarks apply one fixed judge prompt\nacross the whole set. Grading is strictly binary: a response earns one point only if it satisfies every\nstated requirement, and the judge is explicitly instructed not to award partial credit.\n\n## Dataset and licence\n\n1,012 prompts, confirmed directly from the released file: exactly 506 Chinese and 506 English items,\ndivided across the eight instruction types in proportions ranging from 82 items (Counter-Conventional\nFormatting) to 198 (Code without Comments), each type split evenly between the two languages. The paper\nstates the items span 23 domains. No licence is published on the Hugging Face dataset page. There is a\nsingle `train` split used directly for evaluation, with no separate held-out test split.\n\n## Who publishes it\n\nThe paper \"Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?\" was\nsubmitted to arXiv on 4 September 2025, with 21 listed authors headed by Qinyan Zhang, Xinping Lei and\nRuijie Miao; individual institutional affiliations were not confirmed from a source read for this page. The\ndataset is hosted under the m-a-p (Multimodal Art Projection) organisation on Hugging Face, which also\npublishes several other benchmarks and datasets independent of this one.\n\n## Lineage\n\nNo predecessor or successor is tracked for this id in this repository. Its name and its explicit framing as\ntesting the complement of ordinary instruction-following place it alongside IFEval, IFBench and AdvancedIF\n(all in this repository), though none of those pages' own materials cite Inverse IFEval directly, and\nInverse IFEval's own materials read for this page do not cite IFEval by name either -- the relationship is\none of shared territory and an intentionally opposite name, not a confirmed direct lineage. It is also a\nclose cousin of `ifevalcode` (also in this repository), which decouples code correctness from instruction\nadherence rather than testing adherence against a model's trained default behaviour specifically.\n\n## Saturation and contamination\n\nThe paper's abstract states that testing \"existing leading LLMs\" demonstrates the benchmark's necessity,\nwhich implies meaningful failure rates among current models, but this page's research did not extract the\npaper's full results tables, so no specific model score is recorded here. Contamination risk is medium: the\ndataset, including every item's detailed reference criteria, has been public for about a year as of this\nresearch. Grading is a qualitative LLM-judge match rather than an exact-string answer key, so classic\nmemorisation does not apply directly, but each item's expected response pattern is fixed and specific\nenough that a model could plausibly be tuned against the known items and what they expect.\n\n## How to run it\n\nOpenCompass implements this as 16 separate dataset entries, one per instruction type per language, named\n`InverseIFEval_{language}_{type}` (for example `InverseIFEval_en_QC`), built from the config module\n`InverseIFEval_rawprompt_gen.py`. Each entry is scored through OpenCompass's `GenericLLMEvaluator` using\nthat specific item's own bundled judge system prompt and judge prompt template rather than one fixed\ndataset-wide judge configuration, so reproducing a reported score requires using the exact per-item judge\nprompts shipped with the dataset, not a generic LLM-judge setup substituted in by a different harness.\n\n## Reading the numbers\n\nA high score shows a model can suppress its trained default behaviour on command -- answering \"no correct\noption\" instead of guessing, omitting comments when told to, following a formatting instruction it was\nnever trained to prefer. It says nothing about how well the model performs when instructions are ordinary\nand cooperative, which is what IFEval-style benchmarks measure instead; a model can score well on one and\npoorly on the other, since they test opposite failure modes. Because grading depends on a per-item bundled\njudge prompt rather than one fixed judge, scores computed with a substituted judge setup are not a reliable\ncomparison against scores computed with the dataset's own judge prompts.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "inverseifeval",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "InverseIFEval"
  ],
  "category": "instruction-following",
  "contamination": {
   "note": "The dataset, including every item's detailed reference criteria, has been public on Hugging Face for about a year as of this research. Grading is a qualitative LLM-judge match against those criteria rather than an exact-string answer key, so classic answer memorisation does not apply directly -- but because each item's expected response pattern is fixed and specific (for example, \"state that no option is correct\" for a given Question Correction item), a model could still be tuned on the known items and their expected behaviour pattern, which is a narrower but real contamination path.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "English",
    "Chinese"
   ],
   "license": "",
   "modalities": [
    "text",
    "code"
   ],
   "public_test_set": true,
   "size": 1012,
   "size_note": "1,012 prompts, confirmed directly from the released dataset file: exactly 506 Chinese and 506 English items, split across eight instruction types (Code without Comments 198, Deliberately Incorrect Answers 186, Instructional Induction 154, Mid-turn Instruction Modification 108, Counterfactual Answering 108, Question Correction 90, Intentional Textual Flaws 86, Counter-Conventional Formatting 82), each type split evenly in half between the two languages. The paper states the 1,012 items cover 23 domains. OpenCompass's own config reproduces the same eight per-type counts as combined Chinese+English totals, built into 16 separate dataset variants -- one per instruction type per language.\n",
   "splits": "single 'train' split (1,012 rows) used as the evaluation set",
   "url": "https://huggingface.co/datasets/m-a-p/Inverse_IFEval"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 6, slice D",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "InverseIFEval_gen",
   "other": "OpenCompass's config builds 16 separate dataset entries named `InverseIFEval_{language}_{type}` (for example `InverseIFEval_en_QC`), one per instruction type per language, each scored through OpenCompass's `GenericLLMEvaluator` using that specific item's own bundled `judge_system_prompt` and `judge_prompt_template` fields rather than one fixed judge template applied dataset-wide.\n"
  },
  "id": "inverseifeval",
  "last_updated": "2025-09",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "Inverse IFEval tests a model's ability to override the standardised habits instilled by supervised fine-tuning and comply with an instruction that deliberately conflicts with them, rather than testing whether it follows ordinary, cooperative instructions the way IFEval (in this repository) does. Its 1,012 prompts span eight challenge types: Question Correction (recognise that a multiple-choice question has no correct option, instead of picking the closest wrong one), Intentional Textual Flaws, Code without Comments, Counter-Conventional Formatting, Deliberately Incorrect Answers, Instructional Induction, Mid-turn Instruction Modification, and Counterfactual Answering. One released example asks a model to solve a rigged word problem whose four multiple-choice options are all numerically wrong; earning credit requires stating that none of the options is correct rather than defaulting to picking one, the trained habit the item is built to surface.\n",
  "metric": {
   "baseline_note": "Grading is strictly binary per the dataset's own bundled judge system prompt: a response earns one point only if it satisfies every requirement in the item's reference criteria, and zero otherwise, with explicit instructions to the judge not to award partial credit for meeting some but not all requirements. No random or human baseline applies to open-ended generation graded this way.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "binary pass rate under LLM-as-judge grading (0 or 1 per response, scored against the item's own bundled reference criteria), reported overall and per instruction type and language",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "Inverse IFEval",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2509.04292",
   "title": "Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?",
   "url": "https://arxiv.org/abs/2509.04292",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Qinyan Zhang",
    "Xinping Lei",
    "Ruijie Miao",
    "Yu Fu",
    "Haojie Fan",
    "Le Chang",
    "Jiafan Hou",
    "Dingling Zhang",
    "Zhongfei Hou",
    "Ziqiang Yang",
    "Changxin Pu",
    "Fei Hu",
    "Jingkai Liu",
    "Mengyun Liu",
    "Yang Liu",
    "Xiang Gao",
    "Jiaheng Liu",
    "Tong Yang",
    "Zaiyuan Wang",
    "Ge Zhang",
    "Wenhao Huang"
   ],
   "org": "Hosted under the m-a-p (Multimodal Art Projection) organisation on Hugging Face; individual authors' institutional affiliations were not confirmed from a source read for this page",
   "url": "https://huggingface.co/datasets/m-a-p/Inverse_IFEval"
  },
  "released": "2025-09",
  "repo_url": "https://huggingface.co/datasets/m-a-p/Inverse_IFEval",
  "saturation": {
   "as_of": "",
   "note": "The paper's abstract states that experiments on \"existing leading LLMs demonstrate the necessity\" of the benchmark, implying non-trivial failure rates, but the full results tables were not extracted from a source read for this page (the PDF's results tables did not render through the tools available in this research pass), so no specific model score is recorded here.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?",
    "url": "https://arxiv.org/abs/2509.04292"
   },
   {
    "accessed": "2026-09-08",
    "title": "m-a-p/Inverse_IFEval metadata, Hugging Face API",
    "url": "https://huggingface.co/api/datasets/m-a-p/Inverse_IFEval"
   },
   {
    "accessed": "2026-09-08",
    "title": "Inverse_IFEval split info and features, datasets-server",
    "url": "https://datasets-server.huggingface.co/info?dataset=m-a-p/Inverse_IFEval"
   },
   {
    "accessed": "2026-09-08",
    "title": "Inverse_IFEval_Dataset.json: the 1,012 released items",
    "url": "https://huggingface.co/datasets/m-a-p/Inverse_IFEval/resolve/main/Inverse_IFEval_Dataset.json"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass InverseIFEval_rawprompt_gen.py: instruction-type counts and dataset build",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/InverseIFEval/InverseIFEval_rawprompt_gen.py"
   }
  ],
  "status": "active",
  "subcategory": "counter-conventional instruction following: overriding SFT-trained habits to comply with instructions that conflict with them",
  "summary": "Inverse IFEval tests whether a model can override trained habits -- always answering, always correct, always-commented code -- to comply with instructions that deliberately conflict with them.",
  "tags": [
   "instruction-following",
   "counter-conventional",
   "bilingual",
   "llm-judge",
   "adversarial-instructions"
  ],
  "task_format": "Single-turn free-text generation for most items; Mid-turn Instruction Modification introduces a change to the instruction partway through a multi-part prompt instead. Every item carries its own reference criteria describing exactly what a compliant response must and must not contain, and its own bundled LLM-judge system prompt and prompt template, rather than one fixed judge prompt applied across the whole dataset. Items were constructed through a human-in-the-loop pipeline across 23 domains, in matched Chinese and English versions.\n"
 }
}