A 3,500-item BIG-bench task that asks for the unique checkmate-in-one move in standard algebraic notation after a Lichess game prefix.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | BIG-bench SAN checkmate-in-one from Lichess games (3,500 positions) |
| Page status | unknown |
| Metric | exact_str_match |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 3500 |
| Dataset licence | Apache-2.0 |
| Publisher | Google (BIG-bench collaboration); task author at Salesforce |
checkmate_in_one feeds a sequence of chess moves in standard algebraic notation (SAN) such that a checkmate exists on the next move, and asks the model to produce that move. Nitish Shirish Keskar built the set from high-Elo Lichess games that end in mate, then kept only positions with a unique mating move. The probe is whether a language model can keep a board in SAN and finish a forced mate, not whether it plays full games.
Preferred metric exact_str_match with stop_string "#". task.json also lists multiple_choice_grade over legal SAN moves for that position (one option scored 1.0). task_prefix asks for a checkmate-in-one move. append_choices_to_input is false. Canary GUID embedded.
No model card in ModelSpec reports this benchmark yet.