IFEval

IFEval scores whether a model's response obeys objectively checkable instructions, such as word counts or keyword frequency, using code rather than human or LLM judgment.

Also known as: Instruction-Following Eval, Instruction-Following Evaluation for Large Language Models

unassessed

What it measures

IFEval tests whether a model follows explicit, machine-checkable instructions layered onto a prompt, such as "write more than 400 words," "mention the keyword 'AI' at least 3 times," or "wrap your answer in double quotation marks." Each of the roughly 500 prompts combines one or more of 25 instruction types covering things like length, keyword use, formatting, punctuation and casing. Because every instruction can be checked by a program rather than a human or another model, scoring needs no subjective judgment call, unlike most instruction-following or helpfulness evaluations.

Task format

A model is given a prompt containing one or more verifiable instructions and generates a free-form response in a single turn, zero-shot, with no few-shot examples. A separate verification function checks the response against each instruction mechanically (for example, counting words or scanning for a keyword) and returns pass/fail per instruction.

Models reporting this benchmark

Showing the top 200 of 354.

Data

This page as JSON · Edit on GitHub