{
 "body": "\n## What it measures\n\nSpelling Bee reproduces the New York Times Spelling Bee puzzle: a model is given seven letters and\nmust produce as many valid English words of five or more letters as it can, reusing letters freely\nbut never using a letter outside the given seven. It targets rule-following under a combinatorial\nconstraint together with letter-level awareness, a skill subword tokenization can obscure, since a\nmodel's tokens do not map cleanly onto individual letters. A multi-round variant also checks whether\na model can incorporate its own previously confirmed answers as it keeps generating.\n\n## How it is scored\n\nBIG-bench uses a custom point scheme taken directly from the real game: a four-letter word earns 1\npoint, a word of five or more letters earns points equal to its letter count, and a pangram (a word\nusing all seven given letters) earns a 7-point bonus on top. Scores are normalised to each puzzle's\nown maximum possible score and averaged across the 2,000 generated puzzles. The task's README\nreports that the small GPT-2-family and OpenAI-GPT models evaluated when the task was built failed\nto produce any correct words at all, i.e. scored at or near the floor; no human baseline is given.\n\n## Dataset and licence\n\nThe task comprises 2,000 free-text seven-letter puzzles and zero multiple-choice items. Its word\nlist was built by combining Wiktionary word-frequency lists, the standard Unix words file, and\nProject Gutenberg word lists, keeping a word only if it appeared in at least two of the three\nsources, a filter meant to balance common vocabulary against overly obscure or technical terms.\nBecause Wiktionary content is licensed CC BY-SA 3.0 and GFDL, the word list inherits those terms;\nthe puzzles themselves (which seven letters are drawn, and in what combination) are BIG-bench's own\ngenerated content.\n\n## Who publishes it\n\nThe task was contributed to BIG-bench by Ethan Dyer as part of the broader, multi-author BIG-bench\ncollaboration coordinated by Google researchers. No separate standalone paper describing this task\nwas found; its documentation lives in the task's own README and task.json in the BIG-bench\nrepository.\n\n## Lineage\n\nSpelling Bee is one of several hundred independent BIG-bench tasks, modelled directly on the\ncommercial New York Times Spelling Bee puzzle rather than derived from an earlier NLP benchmark. It\nhas no predecessor, successor or variant tracked in this repository.\n\n## Saturation and contamination\n\nNo maintained public leaderboard was found that reports this task in isolation, so its current\nsaturation status among modern models is not established from a source read for this page; the only\ndocumented results are near-floor scores for GPT-2-era models at task creation. The task and its\nfull word list have been publicly downloadable since 2021, and while the README notes a canary GUID\nintended to support exclusion from future training data, that convention only protects against\ncrawlers that honour it, so contamination risk is assessed as medium rather than confirmed high or\nlow.\n\n## How to run it\n\nRun as the `spelling_bee` task in the BIG-bench repository\n(`bigbench/benchmark_tasks/spelling_bee`). No other harness (lm-evaluation-harness, inspect_evals,\nHELM, OpenCompass) implementation was found for this page. Because scoring depends on checking\ngenerated strings against the task's own word list and computing pangram bonuses, exact scores can\ndiffer across re-implementations that use a different or updated word list.\n\n## Reading the numbers\n\nA strong score shows a model can enumerate valid words under a hard letter-set constraint and\nrecognise when it has found every letter in a pangram, a fairly mechanical constraint-satisfaction\nskill rather than broad linguistic reasoning. Because the earliest tested models scored at or near\nzero, apparent gains mostly reflect basic competence at the task's constraints rather than\ndiminishing headroom near a ceiling; with no maintained leaderboard, compare scores only when they\nwere computed against the same word list and scoring implementation.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "spelling_bee",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "reasoning",
  "contamination": {
   "note": "The task file and its word list have been publicly downloadable in the BIG-bench GitHub repository since 2021; the README documents a canary GUID intended to let maintainers exclude the task from future training corpora, but that only works for datasets that respect the canary convention, so exposure in general web-scale training data cannot be ruled out.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Word list drawn from Wiktionary (CC BY-SA 3.0 and GFDL), the Unix words file, and Project Gutenberg word lists; a word is included only if it appears in at least two of the three sources.",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 2000,
   "size_note": "2,000 free-text queries (seven-letter puzzles), confirmed from the task's own README and task.json in the BIG-bench repository; 0 multiple-choice questions.\n",
   "splits": "Single BIG-bench task file of 2,000 generated puzzles; no separate train/validation/test split",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/spelling_bee"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Claude Sonnet 5, sonnet-batch-004 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, sonnet-batch-004"
  },
  "harness": {
   "bigbench": "spelling_bee",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "spelling_bee",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "Spelling Bee presents a model with seven letters (one of them designated as required) and asks it to produce as many valid English words of five or more characters as it can, using only those seven letters and reusing letters freely, in the style of the New York Times Spelling Bee puzzle. It probes rule-following under a combinatorial constraint, tokenization-level letter awareness (a known weak point for subword-tokenized models), and, in its multi-round variant, whether a model can use its own prior correct answers as it keeps generating.\n",
  "metric": {
   "baseline_note": "The task's own README reports that the GPT-2-family and OpenAI-GPT models tested at task creation \"failed to identify any correct words,\" i.e. scored at or near the floor; no human baseline figure is given in the source read for this page.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1.0,
   "name": "Custom point score: 1 point for a 4-letter word, letter-count points for 5+ letter words, +7 bonus for a pangram (all seven letters used), normalised to each game's maximum and averaged across games",
   "random_baseline": 0.0,
   "unit": "normalized score"
  },
  "name": "Spelling Bee",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [
    "Ethan Dyer"
   ],
   "org": "Google (BIG-bench collaboration); task author Ethan Dyer",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/spelling_bee"
  },
  "released": "2021",
  "repo_url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/spelling_bee",
  "saturation": {
   "as_of": "",
   "note": "No maintained public leaderboard for this task in isolation was found. The README documents near-floor performance (essentially no correct words) for the small GPT-2-era models tested at task creation; whether later, larger models saturate the task is not established from a source read for this page.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench spelling_bee task directory (README: rules, scoring, word-list sources and licence, canary GUID, author, baseline GPT-2 results)",
    "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/spelling_bee"
   }
  ],
  "status": "active",
  "subcategory": "constrained word generation: list valid words using only a given set of letters",
  "summary": "A BIG-bench task modelled on the New York Times Spelling Bee puzzle: given seven letters, list as many valid English words over four letters as possible, scored by a pangram-weighted point system.",
  "tags": [
   "reasoning",
   "constrained-generation",
   "tokenization",
   "big-bench"
  ],
  "task_format": "Free-response generation over multiple interactive rounds: the model is given seven letters and must output real words built only from them, with repetition of letters allowed; previously confirmed answers can be fed back in for later rounds. BIG-bench records this as 2,000 free-text queries and zero multiple-choice questions.\n"
 }
}