{
 "body": "\n## What it measures\n\nCaLM is a causal-reasoning suite, not a single QA format. Tasks sit on the causal ladder (association, intervention, counterfactual) and on causal discovery. Items are English or Chinese, and they may be everyday stories, symbol strings, or probability word problems. One counterfactual slice reuses BIG-bench causal-judgement stories.\n\nOpenCompass `calm` is CaLM Lite. Lite keeps 100 items per English task row (and the same in Chinese), 9,200 items in total. The paper\u2019s full set is 126,334 samples. A Lite average is not a full-CaLM number.\n\n## How it is scored\n\nOpenCompass generates an answer, then CaLMEvaluator computes Accuracy with a per-task labeller (choice, probability, or open-ended). Error analysis can be switched on except for the E-CARE generation task. The authors\u2019 evaluator in their own repo is the place to get the paper\u2019s extra metrics. Default OpenCompass prompting is basic zero-shot `{question}`; other prompt styles live in the CaLM repository.\n\n## Dataset and licence\n\nLite: 9,200 public JSON items (4,600 per language), Apache-2.0 on OpenCausaLab/CaLM. Full CaLM is 126,334. Several slices start from older corpora (COPA, E-CARE, CRASS, causal judgement). Probability and symbolic items are described as constructed for CaLM. Download for OpenCompass is the v1.0.0.lite zip from the GitHub releases.\n\n## Who publishes it\n\nSirui Chen, Bo Peng, Meiqi Chen, Ruiqi Wang, Mengying Xu, Xingyu Zeng, Rui Zhao, Shengjie Zhao, Yu Qiao and Chaochao Lu released the technical report on 2024-05-01 (arXiv:2405.00622). OpenCausaLab hosts the site, leaderboards, and code. Lite landed on OpenCompass on 2024-08-08 according to the repository news blurb.\n\n## Lineage\n\nCaLM is not [CHARM](charm.md) (Chinese commonsense). It is also not the [causal_judgment](causal_judgment.md) BIG-bench task, though Lite includes a 100-item `AC-B_causal_judgement_*` slice of that family. COPA appears as a pairwise discovery subset; the SuperGLUE COPA page is a different harness path.\n\n## Saturation and contamination\n\nThe 2024 paper is a large 28-model study, not a current Lite leaderboard snapshot. Files are public, and reused older tasks can leak. Treat contamination as slice-dependent rather than one bit for the whole suite.\n\n## How to run it\n\nOpenCompass: `python run.py --models YOUR_MODEL --datasets calm` (add `--summarizer calm` for a compact mean). Configs live in `opencompass/configs/datasets/calm/`. Each dataset abbr looks like `calm_PCD-B_E-CARE_EN`. The authors\u2019 CLI uses `-l` / `--lite_version` for the same 9,200-item set with more prompt choices. Do not compare a basic-prompt Lite score to a nine-adaptation full-CaLM table.\n\n## Reading the numbers\n\nA strong Lite mean says the model handles this 92-target, two-language sample under basic prompts. It does not say the model can do causal discovery in the wild, or that it matched the paper\u2019s 28-model, 9-adaptation protocol. Report language (EN vs CN), prompt style, and Lite versus full. If the interesting claim is actual causality in stories, look at the causal-judgement slice rather than the headline average.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "calm",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "CaLM Lite",
   "Causal Evaluation of Language Models"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "Lite and full files are public (Apache-2.0). Several targets reuse older public sets (COPA, E-CARE, CRASS, and BIG-bench causal judgement). Exact leakage risk therefore varies by slice. No memorisation study was opened here.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en",
    "zh"
   ],
   "license": "Apache-2.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 9200,
   "size_note": "CaLM Lite: 4,600 English + 4,600 Chinese = 9,200, with 100 items per English row of the Lite table (46 causal tasks \u00d7 2 languages). Full CaLM is 126,334 samples (63,167 per language). OpenCompass README and documents/calm-lite.md both state these figures. One Lite row reuses BIG-bench causal judgement at 100 items (full CaLM 187).\n",
   "splits": "fixed JSON files per task; Lite lives in calm_lite_dataset / data/calm",
   "url": "https://github.com/OpenCausaLab/CaLM"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-029 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-029"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "calm",
   "other": "OpenCompass config opencompass/configs/datasets/calm/calm.py registers abbrs `calm_<task>_EN` and `calm_<task>_CN` (CaLMDataset + CaLMEvaluator). Run: python run.py --models \u2026 --datasets calm [--summarizer calm]. Authors\u2019 repo: python calm/run.py \u2026 -l for Lite. Not CHARM.\n"
  },
  "id": "calm",
  "last_updated": "2024-08",
  "leaderboard_url": "https://opencausalab.github.io/CaLM",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "CaLM tests causal reasoning across Pearl\u2019s ladder: association, intervention, and counterfactuals, plus causal discovery. Each target pairs a causal skill (for example average treatment effect, backdoor adjustment, or actual causality) with a text mode (natural, symbolic, or mathematical) and a language (English or Chinese). Question types include binary classification, choice selection, probability calculation, and one open-ended generation slice. OpenCompass does not run the full 126,334-sample suite by default.\n",
  "metric": {
   "baseline_note": "OpenCompass compute_core_metrics returns Accuracy after task-specific labelling (choice, probability, or open-ended). Optional error analysis is skipped for CEG-O_E-CARE. The paper describes a broader design space of 7 metrics and 12 error types on 28 models; those extra metrics are not the OpenCompass default. No single random or human baseline for the 92 Lite targets was confirmed here.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "Accuracy (OpenCompass CaLMEvaluator core metric; task-specific)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "CaLM (Causal Evaluation of Language Models)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2405.00622",
   "title": "Causal Evaluation of Language Models",
   "url": "https://arxiv.org/abs/2405.00622",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Sirui Chen",
    "Bo Peng",
    "Meiqi Chen",
    "Ruiqi Wang",
    "Mengying Xu",
    "Xingyu Zeng",
    "Rui Zhao",
    "Shengjie Zhao",
    "Yu Qiao",
    "Chaochao Lu"
   ],
   "org": "OpenCausaLab / Shanghai AI Laboratory",
   "url": "https://opencausalab.github.io/CaLM"
  },
  "released": "2024-05",
  "repo_url": "https://github.com/OpenCausaLab/CaLM",
  "saturation": {
   "as_of": "",
   "note": "The May 2024 paper evaluates 28 models on 92 targets \u00d7 9 adaptations (38,910,872 queries claimed). No current OpenCompass-leaderboard top score for CaLM Lite was confirmed from a source opened here.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Causal Evaluation of Language Models (arXiv:2405.00622)",
    "url": "https://arxiv.org/abs/2405.00622"
   },
   {
    "accessed": "2026-09-08",
    "title": "CaLM full text (ar5iv); 126,334 samples, 92 targets",
    "url": "https://ar5iv.labs.arxiv.org/html/2405.00622"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCausaLab/CaLM repository",
    "url": "https://github.com/OpenCausaLab/CaLM"
   },
   {
    "accessed": "2026-09-08",
    "title": "CaLM Apache License 2.0",
    "url": "https://raw.githubusercontent.com/OpenCausaLab/CaLM/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "CaLM Lite documentation (9,200 items)",
    "url": "https://raw.githubusercontent.com/OpenCausaLab/CaLM/main/documents/calm-lite.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass CaLM Lite README",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/calm/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass calm.py dataset list",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/calm/calm.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "CaLMDataset and CaLMEvaluator",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/datasets/calm/calm.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "CaLM project website",
    "url": "https://opencausalab.github.io/CaLM"
   }
  ],
  "status": "active",
  "subcategory": "causal ladder tasks in English and Chinese",
  "summary": "OpenCompass `calm` runs CaLM Lite: 9,200 English and Chinese items over 92 causal targets, a tenth of the full 126,334-sample CaLM suite.",
  "tags": [
   "causal-reasoning",
   "reasoning",
   "bilingual",
   "opencompass"
  ],
  "task_format": "OpenCompass uses zero-shot generation (GenInferencer, max_out_len 500) with prompt template `{question}` and prompt styles `basic` / `basic-CN`. The authors\u2019 own repo supports more adaptations (including in-context variants) and a `--lite_version` flag.\n"
 }
}