{
 "body": "\n## What it measures\n\nUnscramble gives a model a distorted English word and asks it to recover the original spelling.\nThe distortion takes one of five forms: letters cycled around the word, all letters but the first\nand last shuffled (an anagram), all letters but the first two and last two shuffled, random\npunctuation or spaces inserted between every letter, or the word spelled entirely backwards. The\nGPT-3 paper introduced these tasks as a way to probe a narrow, on-the-fly symbolic manipulation\nskill distinct from factual recall: the model must both recognise a target word inside noisy\ncharacter soup and apply a consistent character-level transformation rule it infers from few-shot\nexamples.\n\n## How it is scored\n\nEach of the 5 sub-tasks is scored independently with exact string match between the model's\ngenerated completion and the target word, case- and punctuation-sensitive. There is no partial\ncredit for a near-miss spelling. Because the task is open-ended generation rather than multiple\nchoice, there is no meaningful random baseline; the GPT-3 paper evaluated with 100 in-context\nexamples (K=100) rather than zero-shot, and reporting under a different shot count is not directly\ncomparable.\n\n## Dataset and licence\n\nEach sub-task draws 10,000 examples from the 10,000 most frequent English words, distorted by a\nfixed procedure specific to that sub-task (cycling, partial anagram, random insertion, or\nreversal). The lm-evaluation-harness implementation loads each sub-task from its own config of the\nEleutherAI/unscramble Hugging Face dataset (for example mid_word_1_anagrams for the anagrams1\ntask, cycle_letters_in_word for cycle_letters), evaluating on that dataset's single \"validation\"\nsplit. The Hugging Face dataset card lists its licence as \"other\" without further detail; no more\nspecific licence text was found in the source read for this page.\n\n## Who publishes it\n\nThe task originates from OpenAI's \"Language Models are Few-Shot Learners\" (GPT-3) paper (Brown et\nal., 2020), which introduced it as one of several synthetic and qualitative tasks used to probe\nGPT-3's few-shot abilities. The original data files were released in OpenAI's now-archived\n`openai/gpt-3` GitHub repository. EleutherAI repackaged the data as the EleutherAI/unscramble\ndataset on the Hugging Face Hub and implemented the 5 sub-tasks in lm-evaluation-harness, which is\nthe implementation most current benchmark reports use.\n\n## Lineage\n\nUnscramble has no tracked predecessor or successor benchmark and no page for its 5 individual\nsub-tasks in this repository; they are reported here as parts of one task family rather than as\nseparate pages. It belongs to the same GPT-3 paper section as other synthetic few-shot probes\n(such as arithmetic and word-in-context tasks), none of which are direct variants of it.\n\n## Saturation and contamination\n\nGPT-3 175B's own reported per-task accuracy ranged from about 0.4% on reversed words to about 67%\non random insertion, with cycle letters and the two anagram variants in between, showing the five\nsub-tasks vary enormously in difficulty and were nowhere near saturated at release. No maintained\nleaderboard tracking current frontier-model scores on this specific task set was found for this\npage, so whether harder sub-tasks like reversed words remain difficult for today's models is not\nestablished here. Contamination risk is medium: the source word lists and generation code have\nbeen public since 2020 and are mirrored on the Hugging Face Hub, and because items are generated\ndeterministically from a fixed word-frequency list, a model could score well by having memorised\nthe underlying word list rather than by performing the manipulation.\n\n## How to run it\n\nRun via lm-evaluation-harness with `--tasks unscramble`, which executes all 5 sub-tasks\n(anagrams1, anagrams2, cycle_letters, random_insertion, reversed_words) as independent\ngenerate_until tasks and reports 5 separate exact-match scores rather than one aggregate number.\nThe original GPT-3 paper's own evaluation code and raw data files are in the archived\n`openai/gpt-3` repository. Because the paper's own evaluation used 100-shot prompting, any\nzero-shot or few-shot-with-fewer-examples report from the harness is not directly comparable to\nthe original GPT-3 paper numbers without matching the shot count.\n\n## Reading the numbers\n\nA high exact-match score on the easier sub-tasks (such as random insertion, where the original\nword's letters stay in order) mainly shows a model can strip a fixed kind of noise back out of a\nrecognisable word. A high score on the harder sub-tasks (reversed words, or anagrams with\nfirst/last letters preserved) is a stronger and rarer signal, since GPT-3 175B itself scored under\n1% on reversed words even at K=100. Because the five sub-tasks are reported separately and differ\nso much in difficulty, a single \"unscramble\" number without a breakdown by sub-task tells you\nlittle; look at which sub-tasks are being reported and at what shot count before comparing models.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "unscramble",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Word Scrambling and Manipulation Tasks"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "The underlying word lists and generation scripts have been public since 2020 in the OpenAI GPT-3 repository and are re-hosted on the Hugging Face Hub, so the fixed 10,000-word-per-task item sets could appear in training corpora. Because items are generated deterministically from a fixed frequency-ranked word list rather than curated by hand, memorising the word list (not just the exact scrambled instances) could also inflate scores.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 10000,
   "size_note": "10,000 examples per sub-task (50,000 total across the 5 tasks), drawn from the 10,000 most frequent English words, per the GPT-3 paper (section on synthetic and qualitative tasks). The lm-evaluation-harness configs load each sub-task from a distinct config of the EleutherAI/unscramble Hugging Face dataset (e.g. mid_word_1_anagrams, cycle_letters_in_word, reversed_words) and evaluate on that dataset's \"validation\" split.\n",
   "splits": "Each of the 5 dataset configs exposes a single split used by the harness as its test split, named \"validation\"",
   "url": "https://huggingface.co/datasets/EleutherAI/unscramble"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Claude Sonnet 5, sonnet-batch-008 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, sonnet-batch-008"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "unscramble",
   "opencompass": "",
   "other": "\"unscramble\" is a tag in lm-evaluation-harness grouping 5 separate tasks (anagrams1, anagrams2, cycle_letters, random_insertion, reversed_words), each with its own generate_until config and exact_match metric; running the tag reports all 5 scores rather than a single averaged number.\n"
  },
  "id": "unscramble",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "Unscramble presents a distorted English word and asks the model to produce the original word. The distortion is one of five kinds: letters cycled within the word, all letters but the first and last shuffled, all letters but the first two and last two shuffled, random punctuation or spaces inserted between letters, or the whole word spelled backwards. Solving it requires character-level manipulation on top of knowing the target vocabulary word, a skill the GPT-3 paper framed as a proxy for on-the-fly symbolic pattern recovery rather than a test of world knowledge.\n",
  "metric": {
   "baseline_note": "No random-chance baseline applies since the task is open-ended generation, not multiple choice. No human baseline figure was found in the sources read for this page.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1.0,
   "name": "Exact match (generated word equals target word, case- and punctuation-sensitive)",
   "random_baseline": null,
   "unit": "accuracy"
  },
  "name": "Unscramble",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2005.14165",
   "title": "Language Models are Few-Shot Learners",
   "url": "https://arxiv.org/abs/2005.14165",
   "year": 2020
  },
  "publisher": {
   "authors": [
    "Tom B. Brown",
    "Benjamin Mann",
    "Nick Ryder",
    "Melanie Subbiah"
   ],
   "org": "OpenAI (original task, GPT-3 paper); EleutherAI (dataset repackaging and lm-evaluation-harness implementation)",
   "url": "https://github.com/openai/gpt-3"
  },
  "released": "2020-05",
  "repo_url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/unscramble",
  "saturation": {
   "as_of": "",
   "note": "The GPT-3 paper reported wide variance across the 5 sub-tasks for GPT-3 175B (roughly 0.4% on reversed words to 67% on random insertion), showing the sub-tasks differ hugely in difficulty and were far from saturated in 2020. No maintained leaderboard tracking current frontier-model scores on this exact task set was found for this page, so present-day saturation is not established.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness unscramble task directory (README and 5 task YAML files)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/unscramble"
   },
   {
    "accessed": "2026-09-08",
    "title": "anagrams1.yaml: dataset_path EleutherAI/unscramble, dataset_name mid_word_1_anagrams, generate_until, exact_match, version 2.0",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/unscramble/anagrams1.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "cycle_letters.yaml task config",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/unscramble/cycle_letters.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "reversed_words.yaml task config",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/unscramble/reversed_words.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "Language Models are Few-Shot Learners (GPT-3 paper), Word Scrambling and Manipulation Tasks subsection: 5 task definitions, 10,000 examples per task drawn from the top 10,000 most frequent words, K=100 few-shot, GPT-3 175B per-task accuracy",
    "url": "https://ar5iv.labs.arxiv.org/abs/2005.14165"
   },
   {
    "accessed": "2026-09-08",
    "title": "EleutherAI/unscramble dataset card on Hugging Face Hub (license: other, requires dataset scripting to load)",
    "url": "https://huggingface.co/datasets/EleutherAI/unscramble"
   },
   {
    "accessed": "2026-09-08",
    "title": "openai/gpt-3 repository data directory listing the original .jsonl.gz files for each word-scramble task",
    "url": "https://github.com/openai/gpt-3/tree/master/data"
   }
  ],
  "status": "active",
  "subcategory": "character-level word manipulation (anagrams, letter cycling, reversal, de-noising)",
  "summary": "A battery of 5 character-manipulation tasks from the GPT-3 paper that asks a model to recover an original word from a scrambled, cycled, reversed, or noise-inserted version of it.",
  "tags": [
   "character-manipulation",
   "anagrams",
   "gpt-3",
   "synthetic"
  ],
  "task_format": "Few-shot, open-ended generation: the model is given a scrambled word and must generate the original word, stopping at a newline. lm-evaluation-harness implements it as 5 separate generate_until tasks (anagrams1, anagrams2, cycle_letters, random_insertion, reversed_words) grouped under the \"unscramble\" tag, each scored independently with exact string match against the target completion. The GPT-3 paper evaluated all 5 with 100 in-context examples (K=100).\n"
 }
}