Chess State Tracking

A 6,000-item BIG-bench task that, given a UCI game prefix and a start square, asks for any legal destination square of that piece.

Also known as: State Tracking in Chess, BIG-bench chess_state_tracking

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryreasoning
SubcategoryBIG-bench UCI legal-destination prediction (6 x 1,000 games)
Page statusunknown
Metricexact_str_match
Directionhigher_is_better
Unit%
Dataset size6000
Dataset licenceApache-2.0
PublisherGoogle (BIG-bench collaboration); authors at TTIC

What it measures

chess_state_tracking measures whether a language model has an implicit chess board. The prompt is a Universal Chess Interface (UCI) move list plus the start square of the current non-pawn, non-castling move. The target is any legal destination square for that piece. Authors Shubham Toshniwal, Sam Wiseman, Karen Livescu, and Kevin Gimpel split the work into real Lichess games and synthetic games from a random tree search, each at short, medium, and long length. The skill is legal-move geometry, not mate finding.

Task format

Free-text destination square. Preferred metric exact_str_match with output_regex [a-h][1-8]. Targets are lists of legal squares. task_prefix asks the model to complete the last move by filling in the destination. Canary GUID embedded.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub