MT-Bench

Eighty multi-turn chat questions graded by an LLM judge as a fast, repeatable stand-in for human conversational preference.

Also known as: MT-bench

unassessed

What it measures

MT-Bench gives a chat model 80 open-ended questions spread across eight categories - writing, role-play, extraction, reasoning, math, coding, and two knowledge categories covering STEM and humanities or social science. Each question carries one scripted follow-up, so the model has to hold context across two turns rather than answer a single isolated prompt. The benchmark targets general chat quality and instruction-following on subjective, everyday requests, not narrow factual recall.

Task format

Open-ended two-turn chat completion, graded after generation by a separate judge model rather than by an exact-match answer key.

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub