Programmatic BIG-bench task where a model interactively fills in procedurally generated Sudoku puzzles one cell at a time, scored on command syntax, rule-following and full solution.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | interactive symbolic / logical reasoning |
| Page status | active |
| Metric | combined score |
| Direction | higher_is_better |
| Dataset licence | Apache-2.0 (BIG-bench repository); the puzzle-generation code is also described as drawing on and using Peter Norvig's 'Solving Every Sudoku Puzzle' essay code under Norvig's own permissive license |
| Publisher | Google (BIG-bench collaboration) |
This task gives a model a Sudoku puzzle (4x4, 9x9 or 16x16) rendered as a text grid and asks it to fill empty cells one at a time by issuing repeated '<x> <y> <digit>' commands, receiving the updated board state after each move. It tests whether a model can follow a precisely specified, hard rule set (no repeated digit in a row, column or sub-grid), reason symbolically about a Cartesian coordinate grid, and sustain a multi-step interactive procedure, rather than producing a single free-form answer. The task's author designed it specifically because Sudoku is easy for a basic computer program (e.g. constraint propagation) but was, at the time of authoring, hard for language models, making it a probe of symbolic/algorithmic reasoning rather than pattern-matching over text.
Free-text, multi-turn interactive generation: the model emits one move command per turn (parsed by regex), receives the updated board and any error message, and this repeats until the puzzle's empty cells are exhausted or the context budget is reached.
No model card in ModelSpec reports this benchmark yet.