MastermindEval

lm-eval's MastermindEval tag: six four-way MC tasks that ask for the last remaining Mastermind code after Knuth-style hints.

Also known as: MastermindEval, mastermind_easy, mastermind_hard, mastermind_24_easy, mastermind_24_hard, mastermind_35_easy, mastermind_35_hard, mastermind_46_easy, mastermind_46_hard

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	reasoning
Subcategory	four-way multiple-choice code deduction from pre-played Mastermind games
Page status	active
Metric	accuracy (log-likelihood ranking among four codes)
Direction	higher_is_better
Unit	%
Dataset size	10366
Dataset licence	MIT
Publisher	FLAIR (Humboldt-Universität zu Berlin)

What it measures

MastermindEval, as this lm-eval id, shows a pre-played Mastermind transcript and asks which remaining colour code is the secret. Games were rolled with Knuth's algorithm until one valid code is left. Configurations are 24 (length 2, 4 colours), 35 (length 3, 5 colours), and 46 (length 4, 6 colours). Easy distractors are random codes; hard distractors differ in one symbol. English prompts. This is log-likelihood ranking of four options, not the paper's agentic multi-turn play and not [game24](game24.md).

Task format

lm-eval multiple_choice. Prompt is instruction plus "The secret code is:". Target is the index of answerKey in options.label. Four choices A–D. should_decontaminate true. Tags mastermind, mastermind_easy, mastermind_hard.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub