{
 "body": "\n## What it measures\n\nGaoKaoMATH in OpenCompass is an answer-extraction and answer-checking pipeline for Chinese Gaokao mathematics writeups. The model sees the question, a long response, and the item type, and must pull out the key answer (a letter, a list of letters, or a list of expressions). A judge model then says whether that extraction matches gold.\n\nIt is not [gaokaobench](gaokaobench.md). GAOKAO-Bench scores models as if they were sitting the exam. It is not [chem_exam](chem_exam.md) or [pjexam](pjexam.md).\n\n## How it is scored\n\n`GaoKaoMATHEvaluator` prompts a judge with the question, gold, and candidate, and parses `\\\\boxed{yes}` as correct. Accuracy is 100 \u00d7 correct / n. The public gen config points the judge at Qwen/Qwen2.5-72B-Instruct on a list of private OpenAI-compatible URLs. The README's 95.85% / 95.2% table is extractor accuracy under that judge, not exam solving.\n\n## Dataset and licence\n\nNo public dataset card. The config path is `./data/gaokao_math/test_2k.json`; the README mentions `test_v2.jsonl` and an internal `/cpfs01/...` copy step. Size and licence of the items are not established. OpenCompass code is Apache-2.0. Sample rows in the README are Chinese math items with gold `extract_answer` fields.\n\n## Who publishes it\n\nOpenCompass maintainers. The README names no paper and no authors. No arXiv id was attached to this config. First public appearance date was not established from the files read.\n\n## Lineage\n\nNot a spelling of [gaokaobench](gaokaobench.md) (Zhang et al., Fudan / ECNU, 2,811 exam items). AGIEval's gaokao-mathqa / gaokao-mathcloze splits are a third project. No successor id.\n\n## Saturation and contamination\n\nExtractor accuracy in the mid-90s on this private file does not mean Gaokao math is solved. Contamination of the hidden JSON cannot be judged. If the rows are real past papers, leakage risk is high for any Chinese exam dump; that is a hypothesis, not a verified fact about this file.\n\n## How to run it\n\nCopy the data into `./data/gaokao_math/`, point `GaoKaoMATHEvaluator` at a reachable judge, and run the OpenCompass config `GaoKaoMATH`. Replace the hard-coded `22.8.x.x` URLs. Optional trained 1.5B extractor path in the README is also internal.\n\n## Reading the numbers\n\nA 95% GaoKaoMATH figure is \"the extractor agreed with gold under this judge,\" not \"the model scored 95% on Gaokao math.\" Always name the extractor, the judge, and the missing public file. For exam-solving numbers use [gaokaobench](gaokaobench.md).\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "gaokao_math",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "GaoKao MATH Answer Evaluation",
   "gaokao_math"
  ],
  "category": "math",
  "contamination": {
   "note": "Gold answers live in the local JSON/JSONL. That file is not on a public dataset card. Gaokao math items in general have been circulating in Chinese exam dumps for years, but this page could not inspect the actual rows.\n",
   "risk": "unknown"
  },
  "dataset": {
   "languages": [
    "zh"
   ],
   "license": "OpenCompass Apache-2.0 for the harness; dataset licence not published in the files read",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": null,
   "size_note": "Not established. The runnable config sets path='./data/gaokao_math/test_2k.json'. The folder README tells users to copy an internal tree /cpfs01/shared/public/liuhongwei/data/gaokao_math_dataset/gaokao_math and to replace test_v2.jsonl. Neither filename was found on Hugging Face or in the OpenCompass repo checkout via the public GitHub API. Item count is therefore unknown; \"2k\" in the JSON name is not treated as a verified size.\n",
   "splits": "single local file referenced as test_2k.json or test_v2.jsonl; no public train/test split",
   "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/gaokao_math"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-044 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-044"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "GaoKaoMATH (opencompass/configs/datasets/gaokao_math/gaokao_math_gen_f5fd28.py; type GaoKaoMATHDataset)",
   "other": "Class opencompass.datasets.gaokao_math.GaoKaoMATHDataset / GaoKaoMATHEvaluator. The published gen config hard-codes internal evaluator URLs (http://22.8.x.x:23333/v1) that will not work outside that cluster; the README says to replace model_name and url. No lm-eval, Inspect, or HELM task was found.\n"
  },
  "id": "gaokao_math",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "GaoKaoMATH, as shipped in OpenCompass, is not a contest-math solver. The model under test is given a Gaokao-style mathematics question, a long student-like response, and a question type, and must extract the key answer. Types are \u5355\u9009\u9898 (single choice), \u591a\u9009\u9898 (multiple choice), \u586b\u7a7a\u9898 (fill-in-the-blank), and \u89e3\u7b54\u9898 (worked solution). A second LLM then judges whether that extraction matches the gold `extract_answer`. The config README table reports extractor accuracy of 95.85% for Qwen2.5-72B-Instruct and 95.2% for a 1.5B extractor named gaokao_math_extractor_1.5b_v0.2, both judged by Qwen2.5-72B-Instruct. Chinese prompts are the default in the public gen config.\n",
  "metric": {
   "baseline_note": "README evaluation table (evaluator Qwen2.5-72B-Instruct): 95.85% when that same model is the extractor, 95.2% when gaokao_math_extractor_1.5b_v0.2 is the extractor. No random or human baseline is published. These figures measure extraction-plus-judge agreement, not solving Gaokao math from scratch.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy of extracted answers versus gold extract_answer, LLM-judged",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "GaoKaoMATH",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [],
   "org": "OpenCompass (config and GaoKaoMATHDataset class); no named paper authors were found",
   "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/gaokao_math"
  },
  "released": "",
  "repo_url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/gaokao_math",
  "saturation": {
   "as_of": "",
   "note": "95.85 is the README extractor accuracy for Qwen2.5-72B-Instruct judged by itself, not a frontier math-solving score and not dated beyond the Qwen2.5-era README. Saturation as a math benchmark does not apply; as an extraction task the two reported extractors already sit in the mid-90s.\n",
   "status": "unknown",
   "top_score": 95.85
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass gaokao_math README (task definition, sample rows, 95.85/95.2 table)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/gaokao_math/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "gaokao_math_gen_f5fd28.py (abbr GaoKaoMATH, path test_2k.json, Qwen2.5-72B judge)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/gaokao_math/gaokao_math_gen_f5fd28.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "GaoKaoMATHDataset and GaoKaoMATHEvaluator (boxed yes/no judge, optional extractor)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/datasets/gaokao_math.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass Apache License 2.0",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/LICENSE"
   }
  ],
  "status": "unknown",
  "subcategory": "LLM-as-judge extraction and checking of answers from Gaokao-style math writeups",
  "summary": "OpenCompass LLM-judge pipeline that extracts and checks answers from Gaokao-style math writeups; the item file is not publicly hosted.",
  "tags": [
   "math",
   "chinese",
   "llm-as-judge",
   "answer-extraction",
   "opencompass",
   "gaokao"
  ],
  "task_format": "Zero-shot generation. Input columns: question, response, question_type. Target: extract_answer. Inferencer GenInferencer max_out_len 512. GaoKaoMATHEvaluator calls an OpenAI-compatible judge (config default model_name Qwen/Qwen2.5-72B-Instruct) and looks for \\\\boxed{yes} / \\\\boxed{no}. Optional post-process extractor via with_postprocess and a second URL list. The README also describes a locally trained 1.5B extractor checkpoint that is not on a public Hub path in the files read.\n"
 }
}