IFBench tests whether a model can follow verifiable output constraints it was not trained on, rather than the small fixed set most instruction-following benchmarks reuse.
unassessed
| Category | instruction-following |
|---|---|
| Subcategory | generalization to unseen, verifiable output constraints |
| Page status | active |
| Metric | strict and loose accuracy (prompt-level and instruction-level) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 300 |
| Dataset licence | ODC-BY-1.0 for the dataset (Hugging Face card); the allenai/IFBench code repository is separately licensed Apache-2.0. |
| Publisher | Allen Institute for AI (Ai2), with the University of Washington |
IFBench tests whether a model can follow explicit, machine-checkable output constraints -- things like "answer only yes or no" or "mention the word 'abrakadabra' at least three times" -- when the specific constraints are new to it rather than drawn from a small, well-known set. The paper that introduces it shows that models "strongly overfit on a small set of verifiable constraints from the benchmarks" used to train and evaluate them, and do not generalize well to constraint types they have not seen before; IFBench supplies 58 such new, out-of-domain constraint types, organised into seven categories (count, ratio, words, sentence, format, custom and copy), to measure that generalization gap directly rather than measure compliance with familiar constraints.
A prompt with one or more verifiable constraints appended; the model answers in a single turn or across a short multi-turn exchange where the constraint is introduced separately from the original request. A constraint-specific verification function checks the response programmatically and returns pass or fail, the same mechanical approach IFEval uses.
No model card in ModelSpec reports this benchmark yet.