OpenCompass metric that averages token NLL only on labeled positions, shipped with a local cn-reasoning-val example rather than a fixed public test.
unassessed
| Category | generation |
|---|---|
| Subcategory | token-weighted negative log-likelihood on labeled spans of a reasoning corpus |
| Page status | unknown |
| Metric | average_ppl (token-weighted mean of labeled-span NLL) |
| Direction | lower_is_better |
| Publisher | OpenCompass |
Inference-PPL is a language-modeling score, not a multiple-choice exam. OpenCompass feeds each example's text through InferencePPLOnlyInferencer and asks the model for a tokenwise loss on labeled positions, then averages those losses. The config README says the method is meant for reasoning corpora, where only the "answer" or other marked span should contribute. The bundled example file is named cn-reasoning-val.jsonl under ./data/inference_ppl. The same inferencer can point at other jsonl files with a text field.
Zero-shot completion over raw {text}. The dataset loader requires a local jsonl named {name}.jsonl (default name cn-reasoning-val), drops empty texts, and sets output_column to None. The inferencer then calls get_ppl_tokenwise_from_template(entry, label) and stores per-example ppl and token_len. AverageInferencePPLEvaluator returns sum(ppl) / sum(token_len).
No model card in ModelSpec reports this benchmark yet.