Tests whether a model can produce several genuinely different, still-good answers to the same prompt across repeated samples, rather than near-duplicate outputs; current models fall well short of human writers.
unassessed
| Category | generation |
|---|---|
| Subcategory | output diversity across repeated generations (mode collapse) |
| Page status | active |
| Metric | distinct_k (count of functionally distinct outputs among k generations) and utility_k (patience-weighted cumulative utility of novel outputs) |
| Direction | higher_is_better |
| Unit | count (distinct_k, out of k); utility points (utility_k, reward-model scale 1-10) |
| Dataset size | 1100 |
| Dataset licence | MIT (code repository); the Hugging Face dataset card itself carries no separate data licence tag |
| Publisher | Carnegie Mellon University |
NoveltyBench targets mode collapse: the tendency of a language model to give the same or barely- reworded answer every time it is sampled on a prompt that legitimately has many good answers (write a riddle, name a capital city in Africa, pick a favourite car). It generates k independent completions (10 by default) per prompt across 1,100 prompts designed or filtered to admit multiple valid, meaningfully different responses -- spanning randomness, underspecified factual recall, creative writing, and subjective opinion -- and asks whether the k outputs are actually distinct, useful alternatives rather than trivial rewordings of one underlying answer.
For each prompt, a model produces k independent samples. A fine-tuned classifier groups the k outputs into equivalence classes based on whether a user who saw one would gain anything from seeing another; a separate reward model scores each output's quality. Two metrics are reported per prompt and averaged across the dataset: distinct_k, the count of equivalence classes among the k samples, and utility_k, a patience-weighted cumulative score that credits only the first (highest-quality, earliest-generated) output in each class, discounted by a patience factor for later, still-novel classes.
No model card in ModelSpec reports this benchmark yet.