CaLM (Causal Evaluation of Language Models)

OpenCompass `calm` runs CaLM Lite: 9,200 English and Chinese items over 92 causal targets, a tenth of the full 126,334-sample CaLM suite.

Also known as: CaLM Lite, Causal Evaluation of Language Models

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	reasoning
Subcategory	causal ladder tasks in English and Chinese
Page status	active
Metric	Accuracy (OpenCompass CaLMEvaluator core metric; task-specific)
Direction	higher_is_better
Unit	%
Dataset size	9200
Dataset licence	Apache-2.0
Publisher	OpenCausaLab / Shanghai AI Laboratory

What it measures

CaLM tests causal reasoning across Pearl’s ladder: association, intervention, and counterfactuals, plus causal discovery. Each target pairs a causal skill (for example average treatment effect, backdoor adjustment, or actual causality) with a text mode (natural, symbolic, or mathematical) and a language (English or Chinese). Question types include binary classification, choice selection, probability calculation, and one open-ended generation slice. OpenCompass does not run the full 126,334-sample suite by default.

Task format

OpenCompass uses zero-shot generation (GenInferencer, max_out_len 500) with prompt template `{question}` and prompt styles `basic` / `basic-CN`. The authors’ own repo supports more adaptations (including in-context variants) and a `--lite_version` flag.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub