{
 "body": "\n## What it measures\n\nThe official implementation checks for evidence that a language model was trained on BIG-bench data, by comparing the log-probability the model assigns to a hard-coded canary GUID and to BIG-bench's own git commit hashes against the log-probabilities it assigns to random control strings of the same kind (random UUIDs, and the same hashes reversed). It is a memorization probe, not a knowledge or reasoning task; its own module docstring calls it \"a simple arithmetic language modeling task,\" but that description does not match what the code implements and should be disregarded.\n\n## How it is scored\n\nThe implementation computes `log10_p_dev`: it fits a normal distribution to the control strings' log-probabilities, then reports the log10 of the two-tailed p-value of how far the canary string's log-probability deviates from that distribution, averaged over the GUID check and the git-hash check. A score at or near 0 means no detected deviation; a strongly negative score means the model's probability estimate for the canary is a statistical outlier, which is evidence (not proof) of exposure to that exact string during training.\n\n## Dataset and licence\n\nThere is no dataset of items in the usual sense; the \"test data\" is one hard-coded canary GUID and the BIG-bench repository's own git commit hashes, checked against randomly generated control strings. The source file carries an Apache-2.0 licence header (confirmed directly) and the same canary warning that benchmark data should not appear in training corpora.\n\n## Who publishes it\n\nGoogle BIG-bench maintains the task. No separate paper or current leaderboard was established.\n\n## Lineage\n\nThis is a BIG-bench contamination-detection task. No predecessor or successor was established.\n\n## Saturation and contamination\n\nSaturation is unknown. The task is intended to detect contamination, but no current calibration was established.\n\n## How to run it\n\nRun the BIG-bench task `training_on_test_set` with the official runner and record task revision and model decoding.\n\n## Reading the numbers\n\nA score near 0 means the task found no evidence the model's probability estimate for the BIG-bench canary or its commit hashes was anomalous. A strongly negative score is evidence, not proof, that the model may have memorized that specific string; it does not establish exposure to other benchmark data or general contamination, and a clean score on this task does not rule out contamination elsewhere.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "training_on_test_set",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "reasoning",
  "contamination": {
   "note": "The task explicitly concerns training exposure and carries a canary warning in its source.",
   "risk": "unknown"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "license": "Apache-2.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "The official task source does not state one aggregate item count.",
   "splits": "unknown",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/training_on_test_set"
  },
  "freshness": {
   "luna-new-002": null,
   "luna-new-002 (Codex coordinated)": null,
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review"
  },
  "harness": {
   "bigbench": "training_on_test_set",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "training_on_test_set",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "The task measures whether a model assigns anomalous conditional log-probability to a hard-coded BIG-bench canary GUID and to BIG-bench's own git commit hashes, compared with random control strings, as evidence the model was trained on BIG-bench's public repository.",
  "metric": {
   "baseline_note": "Score is bounded above by 0 (log10 of a p-value of 1, meaning the canary string's log-probability did not deviate from the control strings' distribution). Large negative values indicate the model assigns anomalously high or low probability to the canary GUID or git commit hashes, suggestive of training-data exposure; the source code confirms this directly.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 0,
   "name": "log10 p-value of canary probability deviation (log10_p_dev)",
   "random_baseline": null,
   "unit": ""
  },
  "name": "Training on Test Set",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [],
   "org": "Google BIG-bench",
   "url": "https://github.com/google/BIG-bench"
  },
  "released": "2021",
  "repo_url": "https://github.com/google/BIG-bench",
  "saturation": {
   "as_of": "",
   "note": "No authoritative current leaderboard was established.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench task implementation",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/training_on_test_set/task.py"
   }
  ],
  "status": "active",
  "subcategory": "data contamination detection",
  "summary": "BIG-bench task designed to detect evidence that a language model was trained on benchmark data.",
  "tags": [
   "reasoning",
   "contamination"
  ],
  "task_format": "Conditional log-probability scoring of fixed canary strings against random control strings; the model is not asked to generate text."
 }
}