{
 "body": "\n## What it measures\n\nReordering asks a model to reconstruct order from corrupted text. Words are jumbled, characters and spaces are shuffled, or the model selects the swap sequence needed to build the sentence.\n\nIt isolates sequential structure from broad world knowledge. A model must use syntax and exact order rather than treating a sentence as an unordered bag of keywords.\n\n## How it is scored\n\nBIG-bench uses multiple_choice_grade. The aggregate has three subtasks, but the README does not establish their weighting or one random baseline. Results should identify the subtask and option count.\n\nChoosing an ordered sentence is not equivalent to choosing a swap sequence. Prompt formatting and treatment of spaces affect comparability.\n\n## Dataset and licence\n\nThe generated task header reports 300 multiple-choice dummy-model queries. The README does not state a train/test split or separate dataset licence. The public repository does not settle data licensing.\n\n## Who publishes it\n\nGoogle's BIG-bench repository hosts the task and credits Mukund Varma T. The README cites work on sequential order in language understanding. No current standalone leaderboard was established.\n\n## Lineage\n\nThe task has three named variants: reorder_words, reorder_chars and reorder_byswap. It is related to sentence-ordering tasks but no direct predecessor or successor was established.\n\n## Saturation and contamination\n\nThe generated task and examples are public. A model can be trained on the exact format, and no private refresh policy is described. Saturation is unknown.\n\n## How to run it\n\nRun BIG-bench task undo_permutation. Report aggregate or subtask, exact choice set and whether shuffled spaces are preserved.\n\n## Reading the numbers\n\nA high score means the model reconstructed one synthetic ordering. It does not establish robust syntax understanding on natural text. Compare subtasks separately and pair this task with ordinary language understanding evaluations.\n\nCharacter shuffling is especially sensitive to whitespace and punctuation handling, while word shuffling can be solved with lexical and grammatical cues. The by-swap variant adds a procedural output requirement. These distinctions make aggregate scores useful only when the evaluator reports its subtask mix and exact prompts. They also expose formatting brittleness that ordinary prose benchmarks rarely reveal.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "undo_permutation",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "BIG-bench Undo Permutation",
   "Reordering"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "The synthetic task and canary are public; no private or refreshed test policy is described.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 300,
   "size_note": "The task header reports 300 multiple-choice dummy-model queries.",
   "splits": "reorder_words, reorder_chars and reorder_byswap; no train/test split stated",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/undo_permutation"
  },
  "freshness": {
   "researched": "2026-09-09",
   "researched_by": "GPT-5.6 Luna, luna-new-003 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, luna-new-003"
  },
  "harness": {
   "bigbench": "undo_permutation",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "undo_permutation",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "big_bench",
   "predecessor": "",
   "successors": [],
   "variants": [
    "reorder_words",
    "reorder_chars",
    "reorder_byswap"
   ]
  },
  "measures": "Reordering tests sequential structure through scrambled words, scrambled characters including spaces, and sequences of word swaps.",
  "metric": {
   "baseline_note": "The aggregate combines subtasks with different option structures; one chance baseline is not established.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "multiple_choice_grade accuracy",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "Reordering",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2206.04615",
   "title": "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models",
   "url": "https://arxiv.org/abs/2206.04615",
   "year": 2022
  },
  "publisher": {
   "authors": [
    "Mukund Varma T"
   ],
   "org": "BIG-bench collaboration",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/undo_permutation"
  },
  "released": "2022",
  "repo_url": "https://github.com/google/BIG-bench",
  "saturation": {
   "as_of": "",
   "note": "No current comparative ceiling was established.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-09",
    "title": "BIG-bench Reordering README",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/undo_permutation/README.md"
   },
   {
    "accessed": "2026-09-09",
    "title": "BIG-bench paper",
    "url": "https://arxiv.org/abs/2206.04615"
   }
  ],
  "status": "active",
  "subcategory": "word, character and swap-order reconstruction",
  "summary": "Three synthetic subtasks ask models to reconstruct ordered sentences from scrambled words, characters or swaps.",
  "tags": [
   "big-bench",
   "reasoning",
   "sequence",
   "synthetic"
  ],
  "task_format": "Multiple choice across reorder_words, reorder_chars and reorder_byswap."
 }
}