Four-number puzzles that must equal 24; OpenCompass runs a five-puzzle Tree-of-Thoughts slice of the Princeton Game-of-24 set.
unassessed
| Category | math |
|---|---|
| Subcategory | four-number arithmetic puzzles scored by exact use of each number to make 24 |
| Page status | active |
| Metric | accuracy (share of puzzles whose expression uses the four numbers once and equals 24) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 5 |
| Dataset licence | MIT (Hugging Face card, ToT repository, and ToT LICENSE, Copyright 2023 Shunyu Yao); OpenCompass Apache-2.0 |
| Publisher | Princeton NLP (ToT paper and original task); OpenCompass (harness config) |
Game of 24 is a grade-school puzzle: given four integers, use each exactly once with +, -, *, / to make 24. Yao et al. 2023 (Tree of Thoughts) scrape 1,362 ranked games from 4nums.com and test the hard band indexed 901–1000. Success is a valid expression that equals 24 and uses the four inputs once each. OpenCompass dataset Game24Dataset loads Hugging Face test-time-compute/game-of-24 (or a local CSV) and then slices `data[900:905]`, i.e. five puzzles, not the paper's 100. The OpenCompass config uses ToTInferencer (propose / value / greedy, n_evaluate_sample=3, n_select_sample=5), matching the paper's Game-of-24 ToT setup.
OpenCompass generation with a Tree-of-Thoughts inferencer and Game24PromptWrapper (standard, chain-of-thought, and propose/value prompts copied from the ToT repo). ZeroRetriever. Game24Evaluator checks the extracted expression with sympy and a digit-multiset check. Temperature 0.7 in game24_gen_52a460.py with do_sample=False.
No model card in ModelSpec reports this benchmark yet.