{
 "body": "\n## What it measures\n\n`cmo_fib` asks a model to solve fill-in-the-blank problems from the Chinese Mathematical Olympiad. OpenCompass's short README says the problems cover CMO 2009\u20132022. The model sees the original prompt as the question and must write a final answer, not pick A\u2013D.\n\nThis is contest mathematics in Chinese. It is harder, and differently formatted, than GSM8K-style school word problems. It is not the bilingual multimodal suite [olympiadbench](olympiadbench.md), and it is not AIME.\n\n## How it is scored\n\nOpenCompass uses `MATHEvaluator` version `v2` with `math_postprocess_v2`, the same style of boxed-answer grading used for MATH. Default prompts say, in Chinese, to reason step by step and put the final answer in `\\\\boxed{}`. A second config asks only for the boxed answer. There is no random baseline.\n\nThe README table is the only score sheet opened here. Qwen2.5-Math-72B-Instruct is at 46.15%. Qwen2.5-Math-7B-Instruct is at 42.79%. General Qwen2.5-72B-Instruct is at 20.00% on the same page. The table does not say which hashed config produced those rows.\n\n## Dataset and licence\n\n`CMOFibDataset` reads JSONL, copies `origin_prompt` to `question`, and copies `gold_answer` to `answer`. No n, no split names, and no licence appear in the config folder. Hugging Face `opencompass/cmo_fib` returned 401, so size stays empty. Contest authorship sits with CMO; OpenCompass only packages the eval. OpenCompass code is Apache-2.0; that licence was not verified for this JSONL.\n\nWhether every 2009\u20132022 fill-in item is present, or only a subset, is not stated in the README.\n\n## Who publishes it\n\nOpenCompass (open-compass/opencompass) hosts the task. No separate academic paper for this packaging was found. CMO itself is the Chinese national olympiad. Authors are left empty rather than guessed from unrelated math papers.\n\n## Lineage\n\nThe items are olympiad fill-in blanks, not a new problem campaign. English olympiad pages in this repository include [olympiadbench](olympiadbench.md), [aime](aime.md), and [math](math.md). Those are different contests and different graders. No `cmo` family page exists here. The three OpenCompass configs are prompt variants of one abbr, not three datasets.\n\n## Saturation and contamination\n\n46.15% for a 72B math specialist is still open relative to 100%. General chat models in the same README sit much lower. Past CMO papers are easy to include in pretraining, so a high score can mix memorization with contest skill. OpenCompass does not describe a decontamination filter for this set.\n\n## How to run it\n\nImport `cmo_fib_gen.py` (re-exports `cmo_fib_datasets` from `cmo_fib_gen_ace24b.py`) or pick `cmo_fib_gen_2783e5.py` / `cmo_fib_0shot_notcot_gen_4c6c29.py` explicitly. The Hub path is `opencompass/cmo_fib`. Access may require Hub authentication; this session received 401 without credentials.\n\nName the config hash when you quote a number. CoT versus no-CoT and `max_out_len` 2048 versus the default are enough to move MATH-style scores.\n\n## Reading the numbers\n\nA 46% MATH-evaluator score means the boxed answer often matched `gold_answer` under OpenCompass's matcher. It does not mean the model can write a full olympiad proof, and it does not transfer to English AIME. Because n is unpublished here, do not convert the percentage into an item count. If a report used the non-CoT config, it is not the default `cmo_fib_gen.py` path. Look at [olympiadbench](olympiadbench.md) or [math](math.md) alongside this id if you need a broader contest picture.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "cmo_fib",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "cmo_fib",
   "Chinese Mathematical Olympiad FIB",
   "CMO FIB"
  ],
  "category": "math",
  "contamination": {
   "note": "CMO problems from 2009\u20132022 are widely mirrored in Chinese contest archives and training corpora. Fill-in answers are stored as gold_answer in the JSONL the loader reads. No anti-leakage rewrite is described in the OpenCompass README.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "zh"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "Item count is not in the OpenCompass configs or the short dataset README. The Hub dataset opencompass/cmo_fib returned HTTP 401 (gated or private) from both the API and datasets-server, so no split cardinality was read. Problems are described as CMO 2009\u20132022 fill-in-the-blank items.\n",
   "splits": "loader reads a JSONL path; no split names in CMOFibDataset",
   "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/cmo_fib"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-031 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-031"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "cmo_fib",
   "other": ""
  },
  "id": "cmo_fib",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "cmo_fib is OpenCompass's generation task over fill-in-the-blank problems from the Chinese Mathematical Olympiad (CMO), contest years 2009 through 2022. The model receives the problem text as origin_prompt / question and must produce a final mathematical answer. OpenCompass does not turn the items into multiple choice. It is contest math in Chinese, closer to olympiad fill-in items than to school word problems such as GSM8K.\n",
  "metric": {
   "baseline_note": "Open-ended olympiad blanks have no chance rate. The OpenCompass dataset README table lists Qwen2.5-Math-72B-Instruct at 46.15%, Qwen2.5-Math-7B-Instruct at 42.79%, Qwen2-Math-7B-Instruct at 31.73%, Qwen2-Math-1.5B-Instruct at 23.56%, and internlm2-math-7b at 3.37%, plus lower figures for general Qwen2.5 chat models. Those rows do not name which of the three configs was used.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "MATHEvaluator v2 accuracy after math_postprocess_v2",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "CMO fill-in-the-blank",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [],
   "org": "OpenCompass (dataset packaging); problems from the Chinese Mathematical Olympiad",
   "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/cmo_fib"
  },
  "released": "",
  "repo_url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/cmo_fib",
  "saturation": {
   "as_of": "",
   "note": "OpenCompass README: Qwen2.5-Math-72B-Instruct 46.15% on this packaged set. That is well below 100% and is a math-specialist instruct model, not a dated public leaderboard scrape. The README does not give an as-of month.\n",
   "status": "open",
   "top_score": 46.15
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass cmo_fib config directory",
    "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/cmo_fib"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass cmo_fib README",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/cmo_fib/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "cmo_fib_gen.py re-export",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/cmo_fib/cmo_fib_gen.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "cmo_fib CoT config ace24b",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/cmo_fib/cmo_fib_gen_ace24b.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "cmo_fib CoT config 2783e5",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/cmo_fib/cmo_fib_gen_2783e5.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "cmo_fib 0-shot non-CoT config",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/cmo_fib/cmo_fib_0shot_notcot_gen_4c6c29.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass CMOFibDataset loader",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/datasets/cmo_fib.py"
   }
  ],
  "status": "active",
  "subcategory": "Chinese Mathematical Olympiad fill-in-the-blank problems (2009\u20132022)",
  "summary": "OpenCompass fill-in-the-blank set of Chinese Mathematical Olympiad problems from 2009\u20132022, scored with a MATH-style boxed-answer matcher.",
  "tags": [
   "math",
   "olympiad",
   "chinese",
   "opencompass",
   "fill-in-the-blank"
  ],
  "task_format": "Zero-shot generation. Default configs (cmo_fib_gen_ace24b.py and cmo_fib_gen_2783e5.py) append a Chinese chain-of-thought request and ask for a \\\\boxed{} answer. cmo_fib_0shot_notcot_gen_4c6c29.py skips the \"step by step\" line and only asks for a boxed final answer. Dataset abbr cmo_fib; path opencompass/cmo_fib. Inferencer max_out_len is 2048 in the ace24b and 0-shot configs.\n"
 }
}