{
 "body": "\n## What it measures\n\nUnnatural In-Context Learning tests whether a model infers a transformation when formatting is unusual. It includes identity, date transformations, reversals and two-digit addition where the subtraction symbol stands for addition.\n\nThe examples are synthetic, so the model must attend to local patterns rather than ordinary semantic associations. This is a controlled probe of in-context learning and program induction.\n\n## How it is scored\n\nThe task uses exact completion accuracy over generated examples. Subtasks have different output spaces, so an aggregate must state subtask weighting and shot count. The README reports historical GPT-3 results for selected subtasks rather than a current unified leaderboard.\n\nReport demonstration number and order, delimiters and generation normalization. More shots can help or hurt depending on the transformation and context window.\n\n## Dataset and licence\n\nThe task is programmatically generated. It defines finite distributions for dates, alphabetic strings, common five-letter words and two-digit arithmetic. Reverse subtasks use a public Norvig word list. The header reports 73,420 free-text dummy-model queries but does not present that as a conventional split.\n\nA separate licence for generated examples is not established. The Norvig list has its own source terms.\n\n## Who publishes it\n\nBIG-bench hosts the task, documented in the 2022 BIG-bench paper. Frieda Rong and Percy Liang are the task's credited authors. The task README links to their companion SAIL blog analysis and to the 2020 GPT-3 paper, which motivated the subtask designs but does not itself describe this dataset. No current standalone leaderboard was established.\n\n## Lineage\n\nThe task extends synthetic few-shot experiments described in the GPT-3 paper and SAIL blog. It is not an MMLU subset. Its named subtasks are variants with different input and output distributions.\n\n## Saturation and contamination\n\nThe task is public and algorithmic, so training directly on its distributions could solve it. The authors explicitly say this would defeat the intended experiment. No current saturation or refreshed hidden distribution was established.\n\n## How to run it\n\nRun BIG-bench task unnatural_in_context_learning. Report subtasks separately when possible, preserving shot count, order, format markers and exact matching.\n\n## Reading the numbers\n\nA high score can show that a model inferred a transformation from examples. It does not show broad reasoning or knowledge. Public distributions make memorization possible. Pair it with held-out synthetic transformations and natural-language evaluations.\n\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "unnatural_in_context_learning",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "BIG-bench Unnatural In-Context Learning"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "Generated distributions and task code are public; the authors say direct training on them defeats the experiment.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 73420,
   "size_note": "The task header reports 73,420 free-text dummy-model queries; this is not a stated train/test split.",
   "splits": "aggregate synthetic subtasks; no train/test split stated",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/unnatural_in_context_learning"
  },
  "freshness": {
   "researched": "2026-09-09",
   "researched_by": "GPT-5.6 Luna, luna-new-003 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, luna-new-003"
  },
  "harness": {
   "bigbench": "unnatural_in_context_learning",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "unnatural_in_context_learning",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "big_bench",
   "predecessor": "",
   "successors": [],
   "variants": [
    "identity",
    "dates",
    "dates_unnatural_content",
    "dates_unnatural_form",
    "dates_unnatural_content_and_form",
    "unnatural_addition_2_digit",
    "reverse_natural_content",
    "reverse_to_natural_content"
   ]
  },
  "measures": "Few-shot examples define synthetic transformations; the model must produce the next output. Subtasks cover identity, date formats, reversal and unusual two-digit addition.",
  "metric": {
   "baseline_note": "Subtasks have different output spaces, so one random baseline is not established.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "exact completion accuracy",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "Unnatural In-Context Learning",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2206.04615",
   "title": "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models",
   "url": "https://arxiv.org/abs/2206.04615",
   "year": 2022
  },
  "publisher": {
   "authors": [
    "Frieda Rong",
    "Percy Liang"
   ],
   "org": "BIG-bench collaboration",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/unnatural_in_context_learning"
  },
  "released": "2022",
  "repo_url": "https://github.com/google/BIG-bench",
  "saturation": {
   "as_of": "",
   "note": "Historical GPT-3 analyses do not establish a current aggregate ceiling.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-09",
    "title": "BIG-bench Unnatural In-Context Learning README",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/unnatural_in_context_learning/README.md"
   },
   {
    "accessed": "2026-09-09",
    "title": "SAIL in-context learning analysis",
    "url": "https://ai.stanford.edu/blog/in-context-learning/"
   },
   {
    "accessed": "2026-09-09",
    "title": "GPT-3 few-shot learners paper",
    "url": "https://arxiv.org/abs/2005.14165"
   },
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench paper",
    "url": "https://arxiv.org/abs/2206.04615"
   }
  ],
  "status": "active",
  "subcategory": "few-shot program induction on synthetic formats",
  "summary": "Synthetic identity, date, reversal and arithmetic subtasks test in-context pattern induction outside natural training distributions.",
  "tags": [
   "big-bench",
   "in-context-learning",
   "synthetic",
   "program-induction"
  ],
  "task_format": "Free-text numerical or symbolic completion with variable numbers of demonstrations."
 }
}