OpenCompass LLM-judge pipeline that extracts and checks answers from Gaokao-style math writeups; the item file is not publicly hosted.
unassessed
| Category | math |
|---|---|
| Subcategory | LLM-as-judge extraction and checking of answers from Gaokao-style math writeups |
| Page status | unknown |
| Metric | accuracy of extracted answers versus gold extract_answer, LLM-judged |
| Direction | higher_is_better |
| Unit | % |
| Dataset licence | OpenCompass Apache-2.0 for the harness; dataset licence not published in the files read |
| Publisher | OpenCompass (config and GaoKaoMATHDataset class); no named paper authors were found |
GaoKaoMATH, as shipped in OpenCompass, is not a contest-math solver. The model under test is given a Gaokao-style mathematics question, a long student-like response, and a question type, and must extract the key answer. Types are 单选题 (single choice), 多选题 (multiple choice), 填空题 (fill-in-the-blank), and 解答题 (worked solution). A second LLM then judges whether that extraction matches the gold `extract_answer`. The config README table reports extractor accuracy of 95.85% for Qwen2.5-72B-Instruct and 95.2% for a 1.5B extractor named gaokao_math_extractor_1.5b_v0.2, both judged by Qwen2.5-72B-Instruct. Chinese prompts are the default in the public gen config.
Zero-shot generation. Input columns: question, response, question_type. Target: extract_answer. Inferencer GenInferencer max_out_len 512. GaoKaoMATHEvaluator calls an OpenAI-compatible judge (config default model_name Qwen/Qwen2.5-72B-Instruct) and looks for \\boxed{yes} / \\boxed{no}. Optional post-process extractor via with_postprocess and a second URL list. The README also describes a locally trained 1.5B extractor checkpoint that is not on a public Hub path in the files read.
No model card in ModelSpec reports this benchmark yet.