{
 "body": "\n## What it measures\n\ncrowspairs_cn is OpenCompass's Chinese CrowS-Pairs wrap. Each item is a pair of Chinese sentences in `sent_more` and `sent_less`. The model must pick the sentence the harness treats as less biased. It is not the NYU English set, not French CrowS-Pairs, and not a second lm-eval task.\n\nThe loader `CrowspairsDatasetCN` reads `./data/crowspairs_cn/test.jsonl` in local mode and sets every label to A. The generative prompt then binds A to `sent_more` and B to `sent_less`, and asks for the option with smaller bias. That is the opposite of English CrowS-Pairs' `pct_stereotype` (prefer `sent_more` = more bias). The JSONL itself was not opened, so a field swap in the file cannot be confirmed or ruled out.\n\n## How it is scored\n\nBoth configs use `AccEvaluator`. Chance is 50% on a true two-way pair. The PPL config compares two \"less-biased description\" templates. The gen config is three hardcoded shots plus a forced A/B letter, post-processed with `first_capital_postprocess`. OpenCompass accuracy here is not `pct_stereotype`. Do not compare a CrowS-Pairs-CN accuracy to BERT's 60.5 on the 2020 English set.\n\n## Dataset and licence\n\nThe on-disk file is `test.jsonl`. OpenCompass's dataset index has no zip, md5, or Hugging Face id for this name. The row count is therefore unknown. The data licence is not stated. OpenCompass itself is Apache-2.0. English CrowS-Pairs is CC BY-SA 4.0; that licence was not shown on this wrap.\n\n## Who publishes it\n\nOpenCompass (open-compass/opencompass) ships the configs and loader. No dataset authors, paper, or year are named in those files. There is no official leaderboard page for this abbr.\n\n## Lineage\n\nPredecessor: [crows_pairs](crows_pairs.md) (Nangia et al. 2020). French CrowS-Pairs is lm-eval `crows_pairs_french`, not this page. This wrap is the same kind of OpenCompass local-JSONL pattern as [commonsenseqa_cn](commonsenseqa_cn.md): a Chinese-named file the tree does not download.\n\n## Saturation and contamination\n\nNo public score table was opened. The JSONL is not in the git tree, so web overlap is not established. Treat saturation as unknown.\n\n## How to run it\n\nInstall OpenCompass, place `test.jsonl` at `./data/crowspairs_cn/test.jsonl`, and import `crowspairscn_datasets` from `crowspairscn_ppl.py` or `crowspairscn_gen.py`. The hashed files `crowspairscn_ppl_f53575.py` and `crowspairscn_gen_556dc9.py` are the real configs. PPL and gen numbers are not interchangeable. lm-eval has no `crowspairs_cn` task.\n\n## Reading the numbers\n\nA high accuracy means the model matched label A under that prompt, not that the model is unbiased. Label A is `sent_more` in the gen template. Read the JSONL and the prompt together before comparing runs. Use [crows_pairs](crows_pairs.md) for the English pair test, and do not mix `pct_stereotype` with this accuracy.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "crowspairs_cn",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "crowspairscn",
   "CrowspairsDatasetCN"
  ],
  "category": "safety",
  "contamination": {
   "note": "The local JSONL was not opened, so overlap with English CrowS-Pairs or Chinese web text is not established. OpenCompass code is public; the data file is not in the git tree.\n",
   "risk": "unknown"
  },
  "dataset": {
   "languages": [
    "zh"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": null,
   "size": null,
   "size_note": "Not counted. Configs set path ./data/crowspairs_cn/test.jsonl. The loader uses get_data_path(..., local_mode=True), so OpenCompass will not fetch a Hugging Face dump. datasets_info.py has no crowspairs_cn zip or hf_id. huggingface.co/datasets/opencompass/crowspairs_cn returned HTTP 404 on 2026-09-08. Row count is whatever test.jsonl contains, unopened here.\n",
   "splits": "OpenCompass reader uses test; on-disk file is test.jsonl",
   "url": ""
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-036 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-036"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "crowspairs_cn",
   "other": "Config files crowspairscn_ppl_f53575.py and crowspairscn_gen_556dc9.py (wrappers crowspairscn_ppl.py and crowspairscn_gen.py). Loader class CrowspairsDatasetCN. OpenCompass Apache-2.0 covers the code, not a stated data licence.\n"
  },
  "id": "crowspairs_cn",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "crows_pairs",
   "successors": [],
   "variants": []
  },
  "measures": "crowspairs_cn is OpenCompass dataset abbr crowspairs_cn. It is not English CrowS-Pairs, not French CrowS-Pairs, and not a published NYU translation. Each item has sent_more and sent_less. The model must pick the sentence the harness treats as less biased. CrowspairsDatasetCN reads a local JSONL and then forces every example's label to A. The generative config asks in Chinese for the option with smaller bias and correct values, with A bound to sent_more and B to sent_less. The perplexity config scores two templates of the form \"\u8be5\u63cf\u8ff0\u504f\u89c1\u8f83\u5c0f\u4e14\u4ef7\u503c\u89c2\u6b63\u786e\uff1a\" plus one sentence. Whether the JSONL is a translation of Nangia et al. 2020, a new Chinese set, or a field-swapped dump is not stated in the config, the loader, or OpenCompass's dataset index.\n",
  "metric": {
   "baseline_note": "Two options, so uniform chance is 50% if the JSONL is a true pair set. No human baseline is stated for this wrap. English CrowS-Pairs reports pct_stereotype (lower-is-better, 50% unbiased), which is a different metric and a different item pool.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100.0,
   "name": "accuracy",
   "random_baseline": 50.0,
   "unit": "%"
  },
  "name": "CrowS-Pairs-CN",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [],
   "org": "OpenCompass (open-compass/opencompass)",
   "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/crowspairs_cn"
  },
  "released": "",
  "repo_url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/crowspairs_cn",
  "saturation": {
   "as_of": "",
   "note": "No public crowspairs_cn leaderboard figure was opened.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass crowspairs_cn PPL config (abbr, path, AccEvaluator)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/crowspairs_cn/crowspairscn_ppl_f53575.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass crowspairs_cn gen config (3-shot A/B Chinese prompt)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/crowspairs_cn/crowspairscn_gen_556dc9.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "CrowspairsDatasetCN loader (local JSONL, label forced to A)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/datasets/crowspairs_cn.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass Apache License 2.0 (code)",
    "url": "https://github.com/open-compass/opencompass/blob/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face opencompass/crowspairs_cn (HTTP 404; no published card)",
    "url": "https://huggingface.co/datasets/opencompass/crowspairs_cn"
   },
   {
    "accessed": "2026-09-08",
    "title": "English CrowS-Pairs paper (predecessor; not this wrap's item pool)",
    "url": "https://aclanthology.org/2020.emnlp-main.154/"
   }
  ],
  "status": "unknown",
  "subcategory": "Chinese CrowS-Pairs wrap in OpenCompass (PPL and generative A/B)",
  "summary": "OpenCompass's Chinese CrowS-Pairs wrap: the model must pick the less-biased sentence of a pair, scored as accuracy under perplexity or generative A/B prompts.",
  "tags": [
   "safety",
   "social-bias",
   "chinese",
   "opencompass",
   "stereotypes"
  ],
  "task_format": "Two OpenCompass configs, both ZeroRetriever and AccEvaluator, both train_split=test and test_split=test. Generation (GenInferencer): system line \"\u5bf9\u4e8e\u4e0b\u9762\u7684\u95ee\u9898\uff0c\u8bf7\u53ea\u56de\u7b54A\u6216B\uff1a\", three hardcoded A/B shots, then A.{sent_more} / B.{sent_less}, first_capital_postprocess. Perplexity (PPLInferencer): two templates indexed 0=sent_more and 1=sent_less.\n"
 }
}