{
 "body": "\n## What it measures\n\nIFBench tests whether a model can follow explicit, machine-checkable output constraints -- instructions\nlike \"answer only yes or no\" or \"mention the word 'abrakadabra' at least three times\" -- when the specific\nconstraints are new to it rather than drawn from a small, familiar set. The paper that introduces it shows\nthat models \"strongly overfit on a small set of verifiable constraints from the benchmarks\" used to train\nand evaluate them, scoring well on constraint types they have effectively memorised while failing to\ngeneralize to unseen ones, a skill the authors call precise instruction following. IFBench supplies 58 new,\nhand-curated, out-of-domain constraint types across seven categories (count, ratio, words, sentence,\nformat, custom and copy) specifically to measure that generalization gap, rather than to measure compliance\nwith constraints a model has likely already seen during training or fine-tuning.\n\n## How it is scored\n\nA verification function specific to each constraint checks the model's response programmatically and\nreturns pass or fail, the same mechanical approach IFEval established. IFBench reports both strict accuracy\n(the raw response must satisfy the constraint) and loose accuracy (minor formatting variation, such as\nstray markdown, is tolerated), each computed at the prompt level (every constraint in a prompt must pass)\nand the instruction level (each constraint scored independently). Prompts are run under two conditions: a\nsingle-turn setting, where the constraint is appended to the original request, and a multi-turn setting,\nwhere the constraint is introduced in a separate follow-up turn -- a harder test of whether a model tracks\nan instruction once the immediate context has moved past it.\n\n## Dataset and licence\n\n300 test prompts cover the 58 new constraint types. The dataset card lists the release under ODC-BY-1.0,\ndistinct from the allenai/IFBench code repository's Apache-2.0 licence; both are maintained by the Allen\nInstitute for AI. Separately, and not part of this 300-prompt test set, the authors release 29 additional\nhand-annotated constraints with their own verification functions, plus reinforcement-learning training\nprompts, as material for training models to generalize better rather than for evaluating them.\n\n## Who publishes it\n\nIFBench comes from Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep\nDasigi, Nathan Lambert and Hannaneh Hajishirzi, working at the Allen Institute for AI with the University of\nWashington, published as \"Generalizing Verifiable Instruction Following\" on arXiv in July 2025 and accepted\nto NeurIPS 2025's Datasets and Benchmarks track. Ai2 maintains the reference dataset and code at\ngithub.com/allenai/IFBench; Artificial Analysis independently runs and publishes a continuously updated\nleaderboard.\n\n## Lineage\n\nIFBench is a direct response to IFEval (also in this repository, ifeval.md): the paper builds on IFEval's\nmechanically-verifiable-instruction approach but deliberately uses a disjoint set of 58 new constraints,\nstating these \"go beyond the 25 constraints included in IFEval,\" specifically because models had begun\noverfitting to IFEval's fixed set. It has no successor or variant catalogued in this repository.\n\n## Saturation and contamination\n\nAt release in mid-2025, even strong instruction-tuned models such as Qwen3-32B and Claude 4 Sonnet scored\nbelow 50%, and reinforcement learning with verifiable rewards measurably improved smaller models\n(T\u00fclu-3-8B rose from 28.9% to 45.9%). Artificial Analysis's independently run leaderboard, accessed\n2026-09-08 and tracking 450 models, instead showed Grok 4.3 (medium) leading at 83.3%, with Grok 4.20 0309\n(Reasoning) and MiniMax-M3 close behind at 82.9% each -- a large rise from the paper's results and a top\ntier bunched within half a point, which is why this page treats it as under watch rather than open or\nsaturated. Contamination risk is medium: the test set and verification code have been public for over a\nyear, so specific constraints are plausibly in newer pretraining data, though mechanical, non-answer-matched\nscoring blunts (without eliminating) the reward for memorising one response, and the paper's central finding\n-- that models overfit to constraints they have already seen -- is itself evidence that prior exposure moves\nscores here too.\n\n## How to run it\n\nThe reference evaluation code ships in github.com/allenai/IFBench, using the same per-constraint\nverification-function approach as IFEval, and ties to Ai2's open-instruct RLVR training pipeline used in the\npaper's own experiments. OpenCompass registers the task as `IFBench`, with separate generation and\nraw-prompt config variants. Not confirmed in the lm-evaluation-harness, HELM or BIG-bench task lists.\n\n## Reading the numbers\n\nA high IFBench score is stronger evidence of genuine instruction-following generalization than a high\nIFEval score, precisely because its constraints were built to be unfamiliar rather than reused from a\nwell-known set -- that is the whole point of the benchmark. Because the multi-turn setting is measurably\nharder than the single-turn one (the constraint has to survive being separated from the original request),\ncheck which setting a reported score used before comparing it to another. As with IFEval, a high score says\na model reliably obeys literal, checkable formatting and content rules; it says nothing about whether the\nsubstance of the response is correct or useful, and strict versus loose, and prompt-level versus\ninstruction-level, accuracy can diverge enough to change a ranking.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "ifbench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "instruction-following",
  "contamination": {
   "note": "The 300-prompt test set and its verification code have been fully public on GitHub and Hugging Face since July 2025, over a year before this research pass, so its specific constraints are plausibly present in newer pretraining corpora. As with IFEval, mechanical, non-answer-matched scoring means memorising one plausible response is less directly rewarded than on a fixed-answer benchmark, which may blunt but does not eliminate the risk, and the paper's own finding -- that models overfit to whichever constraints they have already seen -- is itself evidence that prior exposure changes scores.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "ODC-BY-1.0 for the dataset (Hugging Face card); the allenai/IFBench code repository is separately licensed Apache-2.0.",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 300,
   "size_note": "300 test prompts covering 58 new, hand-curated, out-of-domain verifiable constraint types across seven categories, evaluated under both a single-turn setting (the constraint appended to the original prompt) and a multi-turn setting (the constraint introduced in a separate follow-up turn). The paper separately releases 29 additional hand-annotated constraints with verification functions, plus RLVR training prompts, as training material distinct from this 300-prompt test set.\n",
   "splits": "single 300-prompt test set, run under single-turn and multi-turn conditions; no train/validation split in the test dataset itself",
   "url": "https://huggingface.co/datasets/allenai/IFBench_test"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 2, slice F",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "IFBench",
   "other": "The authors' own evaluation code ships in github.com/allenai/IFBench, built on the same verification- function approach as IFEval and tied to Ai2's open-instruct RLVR training pipeline used in the paper's training experiments. OpenCompass registers the task as `IFBench`, with generation and raw-prompt config variants. Not confirmed in the lm-evaluation-harness, HELM or BIG-bench task lists.\n"
  },
  "id": "ifbench",
  "last_updated": "",
  "leaderboard_url": "https://artificialanalysis.ai/evaluations/ifbench",
  "lineage": {
   "family": "",
   "predecessor": "ifeval",
   "successors": [],
   "variants": []
  },
  "measures": "IFBench tests whether a model can follow explicit, machine-checkable output constraints -- things like \"answer only yes or no\" or \"mention the word 'abrakadabra' at least three times\" -- when the specific constraints are new to it rather than drawn from a small, well-known set. The paper that introduces it shows that models \"strongly overfit on a small set of verifiable constraints from the benchmarks\" used to train and evaluate them, and do not generalize well to constraint types they have not seen before; IFBench supplies 58 such new, out-of-domain constraint types, organised into seven categories (count, ratio, words, sentence, format, custom and copy), to measure that generalization gap directly rather than measure compliance with familiar constraints.\n",
  "metric": {
   "baseline_note": "No formal random or human baseline is published. Following IFEval's convention, \"strict\" accuracy checks the raw response and \"loose\" accuracy tolerates minor formatting variation; both are reported at the prompt level (every constraint in the prompt must pass) and the instruction level (each constraint scored on its own).\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "strict and loose accuracy (prompt-level and instruction-level)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "IFBench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2507.02833",
   "title": "Generalizing Verifiable Instruction Following",
   "url": "https://arxiv.org/abs/2507.02833",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Valentina Pyatkin",
    "Saumya Malik",
    "Victoria Graf",
    "Hamish Ivison",
    "Shengyi Huang",
    "Pradeep Dasigi",
    "Nathan Lambert",
    "Hannaneh Hajishirzi"
   ],
   "org": "Allen Institute for AI (Ai2), with the University of Washington",
   "url": "https://github.com/allenai/IFBench"
  },
  "released": "2025-07",
  "repo_url": "https://github.com/allenai/IFBench",
  "saturation": {
   "as_of": "2026-09",
   "note": "At release (mid-2025), the paper reported that even strong instruction-tuned models such as Qwen3-32B and Claude 4 Sonnet scored below 50%. Artificial Analysis's independently run leaderboard, accessed 2026-09-08 and tracking 450 models, instead showed Grok 4.3 (medium) leading at 83.3%, with Grok 4.20 0309 (Reasoning) and MiniMax-M3 tied close behind at 82.9% each -- both a large rise from the paper's own headline finding and a top tier bunched within half a percentage point of each other, which is why this page reads the benchmark as under watch rather than fully open or saturated.\n",
   "status": "watch",
   "top_score": 83.3
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Generalizing Verifiable Instruction Following",
    "url": "https://arxiv.org/abs/2507.02833"
   },
   {
    "accessed": "2026-09-08",
    "title": "Generalizing Verifiable Instruction Following, full text (ar5iv)",
    "url": "https://ar5iv.labs.arxiv.org/html/2507.02833"
   },
   {
    "accessed": "2026-09-08",
    "title": "allenai/IFBench_test dataset card API, Hugging Face",
    "url": "https://huggingface.co/api/datasets/allenai/IFBench_test"
   },
   {
    "accessed": "2026-09-08",
    "title": "allenai/IFBench_test split size, Hugging Face datasets-server",
    "url": "https://datasets-server.huggingface.co/size?dataset=allenai/IFBench_test"
   },
   {
    "accessed": "2026-09-08",
    "title": "allenai/IFBench repository",
    "url": "https://github.com/allenai/IFBench"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass IFBench dataset configs",
    "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/IFBench"
   },
   {
    "accessed": "2026-09-08",
    "title": "IFBench Benchmark Leaderboard, Artificial Analysis",
    "url": "https://artificialanalysis.ai/evaluations/ifbench"
   }
  ],
  "status": "active",
  "subcategory": "generalization to unseen, verifiable output constraints",
  "summary": "IFBench tests whether a model can follow verifiable output constraints it was not trained on, rather than the small fixed set most instruction-following benchmarks reuse.",
  "tags": [
   "instruction-following",
   "verifiable",
   "generalization",
   "zero-shot",
   "multi-turn",
   "text"
  ],
  "task_format": "A prompt with one or more verifiable constraints appended; the model answers in a single turn or across a short multi-turn exchange where the constraint is introduced separately from the original request. A constraint-specific verification function checks the response programmatically and returns pass or fail, the same mechanical approach IFEval uses.\n"
 }
}