A BIG-bench task modelled on the New York Times Spelling Bee puzzle: given seven letters, list as many valid English words over four letters as possible, scored by a pangram-weighted point system.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | constrained word generation: list valid words using only a given set of letters |
| Page status | active |
| Metric | Custom point score: 1 point for a 4-letter word, letter-count points for 5+ letter words, +7 bonus for a pangram (all seven letters used), normalised to each game's maximum and averaged across games |
| Direction | higher_is_better |
| Unit | normalized score |
| Dataset size | 2000 |
| Dataset licence | Word list drawn from Wiktionary (CC BY-SA 3.0 and GFDL), the Unix words file, and Project Gutenberg word lists; a word is included only if it appears in at least two of the three sources. |
| Publisher | Google (BIG-bench collaboration); task author Ethan Dyer |
Spelling Bee presents a model with seven letters (one of them designated as required) and asks it to produce as many valid English words of five or more characters as it can, using only those seven letters and reusing letters freely, in the style of the New York Times Spelling Bee puzzle. It probes rule-following under a combinatorial constraint, tokenization-level letter awareness (a known weak point for subword-tokenized models), and, in its multi-round variant, whether a model can use its own prior correct answers as it keeps generating.
Free-response generation over multiple interactive rounds: the model is given seven letters and must output real words built only from them, with repetition of letters allowed; previously confirmed answers can be fed back in for later rounds. BIG-bench records this as 2,000 free-text queries and zero multiple-choice questions.
No model card in ModelSpec reports this benchmark yet.