{
 "body": "\n## What it measures\n\ngsm8k_contamination measures whether a model treats GSM8K text as more familiar than a matched mock set. OpenCompass scores average perplexity on train problems, test problems, and Skywork's mock file. It does not extract the numeric answer. A model that has seen GSM8K test items should assign them lower PPL than fresh GSM8K-like text. A model that overfit the train split should show a large train\u2013test PPL gap.\n\nThis is not [gsm8k](gsm8k.md) accuracy and not GSM1k. OpenCompass documents a second contamination path (clean vs input-contaminated labels on C-Eval, MMLU, HellaSwag, ARC). That path is a different config family.\n\n## How it is scored\n\n`PPLOnlyInferencer` plus `AveragePPLEvaluator`. Lower average_ppl is the raw metric. Interpretation is comparative: test versus ref, train versus test. Skywork Table 8 uses language-modeling loss on question-plus-answer concatenations and flags outliers in \u03941 = L_test \u2212 L_ref and \u03942 = L_test \u2212 L_train. OpenCompass's example table reports `average_ppl` in a similar numeric range (about 0.5\u20131.6) for 2023-era 7B\u201320B models. Do not treat those example cells as a leaderboard.\n\n## Dataset and licence\n\nGSM8K train and test are OpenAI's public JSONL (7,473 / 1,319, MIT). The reference file is Hugging Face `Skywork/mock_gsm8k_test` (1,415 `text` rows). OpenCompass expects it at `./data/gsm8k-extra/mock_gsm8k_test.jsonl`. Skywork \u00a75.2 and OpenCompass's guide say GPT-4 wrote GSM8K-like samples. The Hugging Face card instead calls the file a mirror of the GSM8K test split. The row count 1,415 does not match 1,319, so the card's \"mirror\" wording is not a literal copy. Mock data uses the Skywork Community License (`license:other`).\n\n## Who publishes it\n\nOpenAI published GSM8K in 2021. Skywork (Kunlun Inc.) described the three-way loss probe in October 2023 (arXiv:2310.19341) and released the mock file. OpenCompass wired that method as `gsm8k_contamination` and cites both the Skywork report and the OpenCompass platform paper. The config directory holds a single file, `gsm8k_contamination_ppl_ecdd22.py`.\n\n## Lineage\n\nPredecessor is [gsm8k](gsm8k.md). [gsm_hard](gsm_hard.md) changes numbers inside GSM8K templates; it is not this PPL probe. GSM1k is a later held-out rewrite for memorization; it has no page here. OpenCompass says this synthetic-reference method currently supports only GSM8K.\n\n## Saturation and contamination\n\nThe probe exists because GSM8K test answers are public and old. A small test-minus-ref PPL gap is the \"clean\" pattern in Skywork Table 8 for several Llama-style models. Large gaps (for example Aquila2-34B \u03941 = \u22120.51 in that table) are the contamination flag. The mock set is itself public, so it can leak into later training runs.\n\n## How to run it\n\nImport `gsm8k_datasets` from `gsm8k_contamination_ppl_ecdd22.py`. Provide `./data/gsm8k` and `./data/gsm8k-extra/mock_gsm8k_test.jsonl`. Report all three abbreviations. Do not mix these PPL values with `gsm8k` exact-match accuracy.\n\n## Reading the numbers\n\naverage_ppl here is not math skill. Read the three numbers together. Test \u226a ref suggests test leakage. Train \u226a test suggests train-set overfitting. Gaps of a few hundredths, as in Skywork's \"almost identical\" Llama-2 row, are not a verdict. Compare only models scored with the same tokenizer and the same concatenation template. For whether a GSM8K accuracy number is trustworthy, use this probe plus a fresh item set, not this probe alone.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "gsm8k_contamination",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "gsm8k_contamination_ppl",
   "gsm8k-train-ppl",
   "gsm8k-test-ppl",
   "gsm8k-ref-ppl",
   "mock_gsm8k_test"
  ],
  "category": "math",
  "contamination": {
   "note": "This id is a contamination detector. GSM8K train and test answers have been public since 2021. The probe's own mock file is public. A low average_ppl on gsm8k-test-ppl relative to gsm8k-ref-ppl is the intended contamination signal, not a capability score.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Mixed: GSM8K MIT; mock_gsm8k_test Skywork Community License (license:other)",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "GSM8K (Cobbe et al.): 7,473 train + 1,319 test, MIT, public answers. Hugging Face datasets-server for Skywork/mock_gsm8k_test: 1,415 test rows, feature text. OpenCompass docs say this is the GPT-4 synthetic GSM8K-style reference from Skywork \u00a75.2. The Hugging Face card for mock_gsm8k_test instead calls it \"a mirror of the GSM8K Test split\" with manually checked answers. 1,415 \u2260 1,319, so it is not a line-for-line copy of openai/gsm8k test. Both descriptions are recorded; the paper's GPT-4 construction is the one OpenCompass's guide cites.\n",
   "splits": "OpenCompass: gsm8k train, gsm8k test, mock ref test (1,415); no hidden hold-out",
   "url": "https://huggingface.co/datasets/Skywork/mock_gsm8k_test"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-046 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-046"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "gsm8k_contamination",
   "other": "Config gsm8k_contamination_ppl_ecdd22.py; docs/en/advanced_guides/contamination_eval.md."
  },
  "id": "gsm8k_contamination",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "gsm8k",
   "successors": [],
   "variants": []
  },
  "measures": "gsm8k_contamination is not a grade-school math test. OpenCompass feeds concatenated question-and-answer text to PPLOnlyInferencer and reports average_ppl on three corpora: the GSM8K training split, the GSM8K test split, and a reference file loaded from ./data/gsm8k-extra/mock_gsm8k_test.jsonl. Skywork (Wei et al., 2023, \u00a75.2) treat the mock set as GSM8K-like text that should not have been in any model's training data. Lower test PPL than mock PPL is read as possible test leakage. Lower train PPL than test PPL is read as possible overfitting on the train split. The skill is distributional familiarity, not solving word problems.\n",
  "metric": {
   "baseline_note": "There is no accuracy number. OpenCompass docs interpret gaps: if test PPL is much lower than ref PPL, the test set may have been in training; if train PPL is much lower than test PPL, the train set may have been overfit. Example table in contamination_eval.md (qwen-7b-hf): train 0.78, test 1.33, ref 1.20. Skywork Table 8 reports language-modeling loss on the same three roles, not OpenCompass average_ppl. Scale of average_ppl versus exp(NLL) was not re-derived from AveragePPLEvaluator.\n",
   "direction": "lower_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "average_ppl",
   "random_baseline": null,
   "unit": ""
  },
  "name": "GSM8K contamination (OpenCompass PPL probe)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2310.19341",
   "title": "Skywork: A More Open Bilingual Foundation Model",
   "url": "https://arxiv.org/abs/2310.19341",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Tianwen Wei",
    "Liang Zhao",
    "Lichang Zhang",
    "OpenCompass Contributors"
   ],
   "org": "OpenCompass (harness); Skywork / Kunlun Inc. (mock set and method); OpenAI (GSM8K)",
   "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/gsm8k_contamination"
  },
  "released": "2023-10",
  "repo_url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/gsm8k_contamination",
  "saturation": {
   "as_of": "",
   "note": "Not an accuracy benchmark. Saturation does not apply. Docs say this OpenCompass path currently supports only GSM8K.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass gsm8k_contamination_ppl_ecdd22.py",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/gsm8k_contamination/gsm8k_contamination_ppl_ecdd22.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass contamination_eval.md (Skywork \u00a75.2 method, example PPL table)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/docs/en/advanced_guides/contamination_eval.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "Skywork technical report arXiv:2310.19341",
    "url": "https://arxiv.org/abs/2310.19341"
   },
   {
    "accessed": "2026-09-08",
    "title": "Skywork full text \u00a75.2 (GPT-4 GSM8K-like reference; Table 8 losses)",
    "url": "https://ar5iv.labs.arxiv.org/html/2310.19341"
   },
   {
    "accessed": "2026-09-08",
    "title": "Skywork/mock_gsm8k_test card (1,415 rows; card text vs paper disagreement)",
    "url": "https://huggingface.co/datasets/Skywork/mock_gsm8k_test"
   },
   {
    "accessed": "2026-09-08",
    "title": "openai/gsm8k (7,473 / 1,319, MIT) as used by GSM8KDataset",
    "url": "https://huggingface.co/datasets/openai/gsm8k"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass GSM8KDataset loader",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/datasets/gsm8k.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass gsm8k_contamination config directory",
    "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/gsm8k_contamination"
   }
  ],
  "status": "active",
  "subcategory": "OpenCompass perplexity comparison of GSM8K train, test, and a mock reference set",
  "summary": "OpenCompass perplexity probe comparing GSM8K train, GSM8K test, and a Skywork mock set to flag training-set overlap, not math accuracy.",
  "tags": [
   "opencompass",
   "contamination",
   "perplexity",
   "gsm8k"
  ],
  "task_format": "Zero-shot perplexity. GSM8K splits use template \"{question} {answer}\" on GSM8KDataset from ./data/gsm8k. The reference uses template \"{text}\" on JsonlDataset. Evaluator AveragePPLEvaluator. Abbreviations gsm8k-train-ppl, gsm8k-test-ppl, gsm8k-ref-ppl. No numeric answer is extracted.\n"
 }
}