AlpacaEval

An automatic, LLM-judged win-rate test of instruction-following that is built and validated to track human preference votes.

Also known as: AlpacaEval 2.0, Length-Controlled AlpacaEval, LC AlpacaEval

unassessed

What it measures

AlpacaEval takes a fixed set of instructions, generates a response from the model under test and from a fixed reference model, and asks a strong LLM judge which response it prefers. The result is a win rate against the reference model rather than an accuracy score on a task with a right answer. It was designed as a fast, cheap stand-in for the kind of human preference voting done by Chatbot Arena, and its authors validate new versions of the metric against correlation with those human votes rather than against a fixed answer key.

Task format

Single-turn instruction in, free-text response out, judged pairwise against a reference model's response to the same instruction by an LLM annotator.

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub