OpenCompass perplexity probe comparing GSM8K train, GSM8K test, and a Skywork mock set to flag training-set overlap, not math accuracy.
unassessed
| Category | math |
|---|---|
| Subcategory | OpenCompass perplexity comparison of GSM8K train, test, and a mock reference set |
| Page status | active |
| Metric | average_ppl |
| Direction | lower_is_better |
| Dataset licence | Mixed: GSM8K MIT; mock_gsm8k_test Skywork Community License (license:other) |
| Publisher | OpenCompass (harness); Skywork / Kunlun Inc. (mock set and method); OpenAI (GSM8K) |
gsm8k_contamination is not a grade-school math test. OpenCompass feeds concatenated question-and-answer text to PPLOnlyInferencer and reports average_ppl on three corpora: the GSM8K training split, the GSM8K test split, and a reference file loaded from ./data/gsm8k-extra/mock_gsm8k_test.jsonl. Skywork (Wei et al., 2023, §5.2) treat the mock set as GSM8K-like text that should not have been in any model's training data. Lower test PPL than mock PPL is read as possible test leakage. Lower train PPL than test PPL is read as possible overfitting on the train split. The skill is distributional familiarity, not solving word problems.
Zero-shot perplexity. GSM8K splits use template "{question} {answer}" on GSM8KDataset from ./data/gsm8k. The reference uses template "{text}" on JsonlDataset. Evaluator AveragePPLEvaluator. Abbreviations gsm8k-train-ppl, gsm8k-test-ppl, gsm8k-ref-ppl. No numeric answer is extracted.
No model card in ModelSpec reports this benchmark yet.