{
 "body": "\nPart of the [CLUE](clue.md) family.\n\n## What it measures\n\nCMNLI asks a model to judge whether a Chinese premise entails, contradicts, or is neutral toward a\nsecond Chinese sentence. Unlike most of this batch, it was not collected in Chinese: the CLUE team\nbuilt it by machine-translating MultiNLI and XNLI, then merging them -- MultiNLI's training data\nbecomes CMNLI's training set; XNLI's dev plus MultiNLI's matched dev, shuffled, becomes CMNLI's dev;\nXNLI's test plus MultiNLI's mismatched test, shuffled, becomes CMNLI's test. No paper, including\nCLUE's own (read in full), documents CMNLI directly; it exists only in the CLUE README, which states\nplainly that this task \"has been replaced on the leaderboard by the native Chinese OCNLI\"\n([clue_ocnli](clue_ocnli.md)), a separate, non-translated dataset built afterward.\n\n## Reading the numbers\n\nThe Hugging Face mirror holds 391,783 training, 12,241 validation and 13,880 test pairs; the CLUE\nREADME instead states 391,782 / 12,426 / 13,880 -- train and test agree, validation differs by 185.\nTest labels are withheld (dummy label of -1 throughout the mirror), so OpenCompass and most current\npapers score the public validation split. No human baseline or current top score was confirmed for\nCMNLI specifically, since CLUE's live leaderboard no longer scores it. Because it is a translated\nstand-in rather than OCNLI's native construction, compare a CMNLI score only against other CMNLI\nscores, not OCNLI's, despite the shared label set.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "clue_cmnli",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "category": "composite",
  "dataset": {
   "languages": [
    "zh"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": 417904,
   "size_note": "391,783 train / 12,241 validation / 13,880 test, per the Hugging Face clue/clue mirror (config cmnli) and its datasets-server split counts. The CLUE GitHub README instead states 391,782 / 12,426 / 13,880 -- train and test agree closely or exactly, but the validation counts differ by 185 examples, a gap this page reports rather than resolves.\n",
   "splits": "train (391,783) / validation (12,241) / test (13,880, label always -1 in the public mirror)",
   "url": "https://huggingface.co/datasets/clue/clue"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 5, slice A"
  },
  "harness": {
   "opencompass": "CLUE_cmnli (CLUE_cmnli_gen / CLUE_cmnli_ppl config variants; loads the opencompass/cmnli-dev mirror)"
  },
  "id": "clue_cmnli",
  "leaderboard_url": "",
  "lineage": {
   "family": "clue",
   "successors": [
    "clue_ocnli"
   ]
  },
  "measures": "CMNLI asks whether a Chinese premise sentence entails, contradicts, or is neutral toward a second Chinese sentence -- three-way natural language inference. It is a translated, adopted task rather than one collected in Chinese: the CLUE team built it by machine-translating two English NLI corpora, MultiNLI and XNLI, then merging them (MultiNLI's training data becomes CMNLI's training set; XNLI's dev plus MultiNLI's matched dev, shuffled, becomes CMNLI's dev; XNLI's test plus MultiNLI's mismatched test, shuffled, becomes CMNLI's test). No paper, including CLUE's own, documents CMNLI directly; it appears only in the CLUE GitHub README.\n",
  "metric": {
   "baseline_note": "33.3% is the three-option random-guess rate. Unlike OCNLI, CMRC2018 and C3, no pre-existing or CLUE-run human-performance figure for CMNLI specifically was found in the paper, the README or the live leaderboard (which no longer scores this task at all).\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 33.3,
   "unit": "%"
  },
  "name": "CLUE: CMNLI (Chinese Multi-Genre NLI)",
  "page_kind": "subset",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [
    "Liang Xu",
    "Hai Hu",
    "Xuanwei Zhang",
    "Lu Li",
    "Chenjie Cao",
    "Yudong Li",
    "Yechen Xu",
    "Kai Sun",
    "Dian Yu",
    "Cong Yu",
    "Yin Tian",
    "Qianqian Dong",
    "Weitang Liu",
    "Bo Shi",
    "Yiming Cui",
    "Junyi Li",
    "Jun Zeng",
    "Rongzhao Wang",
    "Weijian Xie",
    "Yanting Li",
    "Yina Patterson",
    "Zuoyu Tian",
    "Yiwen Zhang",
    "He Zhou",
    "Shaoweihua Liu",
    "Zhe Zhao",
    "Qipeng Zhao",
    "Cong Yue",
    "Xinrui Zhang",
    "Zhengliang Yang",
    "Kyle Richardson",
    "Zhenzhong Lan"
   ],
   "org": "CLUE benchmark team; assembled by machine-translating and merging MultiNLI (Williams et al., New York University) and XNLI (Conneau et al., Facebook AI Research and NYU)",
   "url": "https://github.com/CLUEbenchmark/CLUE"
  },
  "released": "2019-11",
  "repo_url": "https://github.com/CLUEbenchmark/CLUE",
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "CLUEbenchmark/CLUE GitHub repository (README, task 4: CMNLI, states it was replaced by OCNLI on the leaderboard)",
    "url": "https://github.com/CLUEbenchmark/CLUE"
   },
   {
    "accessed": "2026-09-08",
    "title": "CLUE: A Chinese Language Understanding Evaluation Benchmark (Xu et al., arXiv:2004.05986) -- read in full; does not mention CMNLI",
    "url": "https://arxiv.org/abs/2004.05986"
   },
   {
    "accessed": "2026-09-08",
    "title": "A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference (MultiNLI; Williams, Nangia, Bowman)",
    "url": "https://arxiv.org/abs/1704.05426"
   },
   {
    "accessed": "2026-09-08",
    "title": "XNLI: Evaluating Cross-lingual Sentence Representations (Conneau et al.)",
    "url": "https://arxiv.org/abs/1809.05053"
   },
   {
    "accessed": "2026-09-08",
    "title": "clue/clue dataset metadata, Hugging Face API (config cmnli)",
    "url": "https://huggingface.co/api/datasets/clue/clue"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass CLUE_cmnli_gen_1abf97.py config",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/CLUE_cmnli/CLUE_cmnli_gen_1abf97.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "CLUE1.1 leaderboard, cluebenchmarks.com -- CMNLI is not among its scored columns (fetched via scripts/benchmarks/fetch.py)",
    "url": "https://www.cluebenchmarks.com/rank.html"
   }
  ],
  "status": "active",
  "subcategory": "natural language inference (Chinese, machine-translated)",
  "summary": "CLUE's translated NLI task, built from machine-translated MultiNLI and XNLI merged into one set; formally replaced by OCNLI on CLUE's own leaderboard.",
  "tags": [
   "chinese",
   "natural-language-inference",
   "translated",
   "clue-subset"
  ],
  "task_format": "Three-way classification (entailment / neutral / contradiction) over a Chinese sentence pair, scored by accuracy.\n"
 }
}