WildBench

1,024 hard tasks mined from over a million real chatbot conversations, scored automatically by an LLM judge against a task-specific checklist.

Also known as: WildBench v2

unassessed

What it measures

WildBench draws its questions from real conversations logged by AI2's WildChat project rather than writing them by hand, then keeps only the harder, more distinguishing ones. Tasks span writing assistance, coding, math, data analysis, role play and planning, and over a fifth of the conversations run to three or more turns, so the benchmark exercises both single-turn quality and multi-turn coherence. The goal is to approximate what a broad population of real users actually asks chat models to do, rather than a curated academic question set.

Task format

Open-ended chat completion (single- or multi-turn) over a real user task, graded after the fact by an LLM judge using a per-task checklist rather than a fixed answer key.

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub