{
 "body": "\n## What it measures\n\nInference-PPL asks how surprised a model is on the labeled tokens of a text, not which letter it would pick on a quiz. OpenCompass's README describes the intended use as scoring a reasoning corpus where only marked positions count. The default config looks for `cn-reasoning-val.jsonl` under `./data/inference_ppl`. That file is not in the public configs tree, so this page cannot name the prompts or the language. The method can be reused on any jsonl with a `text` field.\n\nThis is not OpenCompass `PPLInferencer`. That class scores multiple-choice items by taking the option with the lowest full-sequence perplexity. Inference-PPL never chooses a label.\n\n## How it is scored\n\n`InferencePPLOnlyInferencer` calls `get_ppl_tokenwise_from_template` and writes per-example `ppl` and `token_len`. `AverageInferencePPLEvaluator` returns `sum(ppl) / sum(token_len)`, a token-weighted mean. The README's displayed formula is mean negative log probability, not `exp` of that mean. Lower is better. The four example scores (0.43\u20130.59) sit in NLL range. Do not treat them as conventional perplexity.\n\nThe example run script evaluates Qwen1.5-7B, Qwen1.5-14B, Llama-2-7B and Llama-2-13B with batch size 8. Those numbers are tied to the unpublished example file.\n\n## Dataset and licence\n\nNo public item count, language list or dataset licence was established. The loader requires a local jsonl. OpenCompass itself is Apache-2.0; that grant covers the harness, not an unseen corpus. `public_test_set` is false because the default file is not on GitHub.\n\n## Who publishes it\n\nOpenCompass maintains the config, inferencer and evaluator. No paper, author list or leaderboard is attached to this dataset directory. The example script lives at `examples/eval_inference_ppl.py`.\n\n## Lineage\n\nThis is a scoring method, not a successor to a named exam. It is easy to confuse with OpenCompass's many `*_ppl` multiple-choice configs (for example Winograd-ppl). Those pick a class. This one reports a mean labeled-span loss.\n\n## Saturation and contamination\n\nThe only numbers in the README are four mid-size open models on the example file. No ceiling is known. Contamination cannot be judged without the corpus.\n\n## How to run it\n\nPlace a jsonl at `./data/inference_ppl/cn-reasoning-val.jsonl`, then:\n\n```\npython run.py examples/eval_inference_ppl.py\n```\n\nor import `inference_ppl_datasets` from `opencompass.configs.datasets.inference_ppl.inference_ppl`. Change `name` to point at another jsonl stem. Compare runs only when the file, the labeled-span definition and the token-weighted average match.\n\n## Reading the numbers\n\nA lower Inference-PPL means the model assigned higher probability to the labeled tokens of that local file. It does not mean the model solved the underlying reasoning items, and it is not a multiple-choice accuracy. Quote the jsonl name and whether the reporter used `exp` of the mean. A 0.43 on Llama-2-13B in the README is not \"43%.\" Pair it with a task that grades answers if you care about correctness rather than token loss.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "inference_ppl",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "inference-ppl",
   "OpenCompass InferencePPL"
  ],
  "category": "generation",
  "contamination": {
   "note": "The example corpus is a local jsonl that is not in the public configs tree. Whether those texts appear in pretraining is not established.\n",
   "risk": "unknown"
  },
  "dataset": {
   "languages": [],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": null,
   "size_note": "OpenCompass does not publish an item count. The default config loads a local file ./data/inference_ppl/cn-reasoning-val.jsonl; that file is not in the GitHub configs tree. samples=None means the full jsonl test split; setting samples to an integer truncates to test[:N].\n",
   "splits": "local jsonl loaded as Hugging Face test (optional test[:N] truncation)",
   "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/inference_ppl"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-050 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-050"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "inference_ppl",
   "other": "Runnable example: python run.py examples/eval_inference_ppl.py. Dataset abbr is inference-ppl. Inferencer InferencePPLOnlyInferencer; evaluator AverageInferencePPLEvaluator in icl_misc_evaluator.py. Distinct from OpenCompass PPLInferencer, which picks a multiple-choice label by minimum sequence perplexity.\n"
  },
  "id": "inference_ppl",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "Inference-PPL is a language-modeling score, not a multiple-choice exam. OpenCompass feeds each example's text through InferencePPLOnlyInferencer and asks the model for a tokenwise loss on labeled positions, then averages those losses. The config README says the method is meant for reasoning corpora, where only the \"answer\" or other marked span should contribute. The bundled example file is named cn-reasoning-val.jsonl under ./data/inference_ppl. The same inferencer can point at other jsonl files with a text field.\n",
  "metric": {
   "baseline_note": "The README writes a mean negative-log-probability formula and labels it ppl; it is not exp(NLL). Example numbers in that README are Qwen1.5-7B 0.59, Qwen1.5-14B 0.54, Llama-2-7B 0.49, Llama-2-13B 0.43, which sit in NLL range rather than typical perplexity (>1). No random or human baseline applies.\n",
   "direction": "lower_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "average_ppl (token-weighted mean of labeled-span NLL)",
   "random_baseline": null,
   "unit": ""
  },
  "name": "Inference-PPL",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [],
   "org": "OpenCompass",
   "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/inference_ppl"
  },
  "released": "",
  "repo_url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/inference_ppl",
  "saturation": {
   "as_of": "",
   "note": "The README table only lists four 2024-era open models on the example file. No current leaderboard cell was read.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "inference_ppl README (labeled-span NLL formula, cn-reasoning-val, four model numbers)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/inference_ppl/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "inference_ppl.py (InferencePPLDataset, abbr inference-ppl, path ./data/inference_ppl)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/inference_ppl/inference_ppl.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "examples/eval_inference_ppl.py (Qwen1.5 7B/14B and Llama-2 7B/13B)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/examples/eval_inference_ppl.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "InferencePPLDataset loader (local jsonl, empty-text filter)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/datasets/inference_ppl.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "InferencePPLOnlyInferencer (tokenwise ppl, no choice)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/openicl/icl_inferencer/icl_inference_ppl_only_inferencer.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "AverageInferencePPLEvaluator (sum ppl / sum token_len)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/openicl/icl_evaluator/icl_misc_evaluator.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass Apache License 2.0",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/LICENSE"
   }
  ],
  "status": "unknown",
  "subcategory": "token-weighted negative log-likelihood on labeled spans of a reasoning corpus",
  "summary": "OpenCompass metric that averages token NLL only on labeled positions, shipped with a local cn-reasoning-val example rather than a fixed public test.",
  "tags": [
   "opencompass",
   "perplexity",
   "language-modeling",
   "reasoning-corpus"
  ],
  "task_format": "Zero-shot completion over raw {text}. The dataset loader requires a local jsonl named {name}.jsonl (default name cn-reasoning-val), drops empty texts, and sets output_column to None. The inferencer then calls get_ppl_tokenwise_from_template(entry, label) and stores per-example ppl and token_len. AverageInferencePPLEvaluator returns sum(ppl) / sum(token_len).\n"
 }
}