{
 "body": "\n## What it measures\n\nOpenCompass `nq_cn` is closed-book short-answer QA with a Chinese wrapper.\nThe model is given a question field inside \"\u95ee\u9898: \u2026?\\n\u7b54\u6848\u662f\uff1a\" and must\nproduce an answer string. There is no Wikipedia passage in the prompt.\n\nThe English OpenCompass task `nq` uses NaturalQuestionDataset, Hugging Face\npath `opencompass/natural_question`, and the prompt \"Question: \u2026?\\nAnswer: \".\n`nq_cn` is a separate class and a local jsonl path. It is not lm-eval\n[nq_open](nq_open.md) and not HELM [natural_qa](natural_qa.md).\n\n## How it is scored\n\n`NQEvaluatorCN` reports a 0-100 score. It takes the first line of the\ncompletion, strips a duplicated \"\u7b54\u6848\u662f\uff1a\" prefix, lowercases, post-processes,\nand requires exact equality with one reference string. English\n`NQEvaluator` instead uses substring containment (`cand in pred`). The config\nis zero-shot (`ZeroRetriever`). The reader\u2019s `train_split` is `test`, so the\nnumbers are on `test.jsonl`. Dev rows collapse answers to the first alias;\ntest rows keep a list.\n\n## Dataset and licence\n\nOn-disk layout is `./data/nq_cn/dev.jsonl` and `test.jsonl`, loaded with\n`local_mode=True`. Count, licence and translator are not in the repository.\nA Hugging Face id `opencompass/nq_cn` was not found. Do not copy English NQ\nsplit sizes here.\n\n## Who publishes it\n\nThe runnable definition is OpenCompass (`open-compass/opencompass`). No\ndataset authors or paper are named on the nq_cn config. The English Natural\nQuestions corpus is Kwiatkowski et al. (TACL 2019); that paper does not\ndescribe this Chinese wrap.\n\n## Lineage\n\nRelated ids in this repository: [nq_open](nq_open.md) (lm-eval English\nclosed-book NQ-Open) and [natural_qa](natural_qa.md) (HELM short-answer NQ).\nOpenCompass English `nq` has no page yet. [HumanEval-CN](humaneval_cn.md) is\nthe same harness pattern (Chinese instruction, separate files), not the same\ntask.\n\n## Saturation and contamination\n\nNo top score was read. Contamination cannot be graded until the jsonl\nprovenance is public. If the items are translated public NQ questions, treat\nscores as leak-prone.\n\n## How to run it\n\nOpenCompass with dataset config `nq_cn` after placing `data/nq_cn/*.jsonl`.\nDo not compare to English `nq`, to lm-eval `nq_open`, or to HELM `natural_qa`\nF1. Name the exact-match rule (equality, not substring).\n\n## Reading the numbers\n\nA high `nq_cn` score means the Chinese completion exactly matched a listed\nanswer after light normalisation. It does not measure retrieval, citation or\ncurrent-events QA. Because the local files were not counted here, always\npublish n and the data revision with the score.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "nq_cn",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "nqcn",
   "NaturalQuestionDatasetCN"
  ],
  "category": "knowledge",
  "contamination": {
   "note": "If the jsonl is a translation of public Natural Questions, leakage risk is high. The files themselves were not opened, so risk is left unknown rather than inferred.\n",
   "risk": "unknown"
  },
  "dataset": {
   "languages": [
    "zh"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": null,
   "size": null,
   "size_note": "Not established. The loader expects ./data/nq_cn/{dev,test}.jsonl and sets local_mode=True, so OpenCompass will not fetch a Hugging Face dump. No public opencompass/nq_cn dataset card was found (huggingface.co returned 404). Dev rows keep only answer[0]; test rows keep the full answer list. File lengths were not available without the local data pack.\n",
   "splits": "dev.jsonl and test.jsonl on disk; OpenCompass reader uses the test split",
   "url": ""
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-013 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-013"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "nq_cn",
   "other": "Config opencompass/configs/datasets/nq_cn/nqcn_gen.py imports nqcn_gen_141737.py. Dataset class NaturalQuestionDatasetCN, evaluator NQEvaluatorCN, path ./data/nq_cn.\n"
  },
  "id": "nq_cn",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "This id is OpenCompass dataset abbr nq_cn, not English OpenCompass nq, not lm-eval nq_open, and not HELM natural_qa. The model sees a Chinese prompt of the form \"\u95ee\u9898: {question}?\\n\u7b54\u6848\u662f\uff1a\" and must emit a short answer string. The loader NaturalQuestionDatasetCN reads local ./data/nq_cn/dev.jsonl and test.jsonl. Whether those questions are translations of English Natural Questions or a new Chinese set is not stated in the config or loader. The reader evaluates the test split (train_split is set to test).\n",
  "metric": {
   "baseline_note": "NQEvaluatorCN lowercases, keeps the first line, strips a leading \"\u7b54\u6848\u662f\uff1a\" if present, runs general_postprocess, then counts an item correct only when a candidate answer string equals the prediction. That is stricter than OpenCompass English NQEvaluator, which counts a candidate as correct if it is a substring of the prediction. No random or human baseline is published for this config.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "exact match after OpenCompass post-processing (NQEvaluatorCN score, 0-100)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "NQ-CN (OpenCompass)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [],
   "org": "OpenCompass (open-compass/opencompass)",
   "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/nq_cn"
  },
  "released": "",
  "repo_url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/nq_cn",
  "saturation": {
   "as_of": "",
   "note": "No public nq_cn leaderboard figure was opened.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass nqcn_gen.py (re-exports nqcn_datasets)",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/nq_cn/nqcn_gen.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "nqcn_gen_141737.py (abbr nq_cn, Chinese prompt, ZeroRetriever)",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/nq_cn/nqcn_gen_141737.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "NaturalQuestionDatasetCN and NQEvaluatorCN (exact equality match)",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/datasets/natural_question_cn.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "English NaturalQuestionDataset / NQEvaluator (substring match contrast)",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/datasets/natural_question.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass English nq config (abbr nq, path opencompass/natural_question)",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/nq/nq_gen_3dcea1.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass datasets __init__ (imports natural_question_cn)",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/datasets/__init__.py"
   }
  ],
  "status": "unknown",
  "subcategory": "Chinese-prompt closed-book short-answer question answering in the Natural Questions style",
  "summary": "OpenCompass Chinese-prompt Natural Questions wrap: zero-shot short-answer generation scored by exact match on local jsonl files.",
  "tags": [
   "question-answering",
   "chinese",
   "opencompass",
   "short-answer",
   "closed-book"
  ],
  "task_format": "Zero-shot generation. ZeroRetriever, GenInferencer. Chinese instruction wrapper around a {question} field. pred_role BOT.\n"
 }
}