A battery of 5 character-manipulation tasks from the GPT-3 paper that asks a model to recover an original word from a scrambled, cycled, reversed, or noise-inserted version of it.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | character-level word manipulation (anagrams, letter cycling, reversal, de-noising) |
| Page status | active |
| Metric | Exact match (generated word equals target word, case- and punctuation-sensitive) |
| Direction | higher_is_better |
| Unit | accuracy |
| Dataset size | 10000 |
| Publisher | OpenAI (original task, GPT-3 paper); EleutherAI (dataset repackaging and lm-evaluation-harness implementation) |
Unscramble presents a distorted English word and asks the model to produce the original word. The distortion is one of five kinds: letters cycled within the word, all letters but the first and last shuffled, all letters but the first two and last two shuffled, random punctuation or spaces inserted between letters, or the whole word spelled backwards. Solving it requires character-level manipulation on top of knowing the target vocabulary word, a skill the GPT-3 paper framed as a proxy for on-the-fly symbolic pattern recovery rather than a test of world knowledge.
Few-shot, open-ended generation: the model is given a scrambled word and must generate the original word, stopping at a newline. lm-evaluation-harness implements it as 5 separate generate_until tasks (anagrams1, anagrams2, cycle_letters, random_insertion, reversed_words) grouped under the "unscramble" tag, each scored independently with exact string match against the target completion. The GPT-3 paper evaluated all 5 with 100 in-context examples (K=100).
No model card in ModelSpec reports this benchmark yet.