{ "body": "\n## What it measures\n\nACLUE asks whether a model understands ancient Chinese. Each item is a four-option question about classical text: word sense, \u901a\u5047 characters, named entities, unpunctuated sentence cuts, couplets, poetry, medical prose, literature, phonetics, or general \u56fd\u5b66. The time span in the paper and dataset card is the Xia dynasty through the Ming dynasty.\n\nThe suite is not CLUE. CLUE is modern Chinese NLU (this repository's `clue` family). It is also not CMMLU, which tests modern Chinese school and professional knowledge. ACLUE is a separate 15-task exam aimed at classical language.\n\n## How it is scored\n\nThe original paper reports zero-shot accuracy, averaged over the 15 tasks, with chance at 25%. Table 1 lists ChatGLM2 at 37.4% and ChatGPT at 36.9%. The GitHub English README's zero-shot table instead lists ChatGLM2-6B at 37.34 and ChatGPT at 36.82. lm-evaluation-harness exposes a group `aclue` that runs all 15 subjects as log-likelihood multiple choice and aggregates `acc` and `acc_norm`, weighted by size. Few-shot uses the five-item development split (`sampler: first_n`). A generative letter-picking protocol and this likelihood protocol are not interchangeable.\n\n## Dataset and licence\n\nThe Hugging Face mirror `tyouisen/aclue` is the copy lm_eval loads. Its per-task Total Q. column sums to 4,967 items. Generated lexical and poetry tasks sit at 500 items; collected knowledge tasks are smaller (phonetics and reading comprehension 101 each). Each task has a five-question development split. Test labels are in the public files.\n\nThe GitHub project is MIT. The dataset is stated as CC BY-NC-SA 4.0 in the GitHub README and in the Hugging Face card body. The Hugging Face YAML header instead says `cc-by-nc-4.0`. This page records CC-BY-NC-SA-4.0 and treats the YAML tag as a conflict, not a second licence.\n\n## Who publishes it\n\nYixuan Zhang and Haonan Li, with Li's affiliation given as MBZUAI on the paper. The work appeared at the Ancient Language Processing Workshop (Varna, September 2023) and as arXiv:2310.09550. Code and data live at `isen-zhang/ACLUE`. The only leaderboard found is the static zero-shot table in that README.\n\n## Lineage\n\nACLUE is a standalone ancient-Chinese suite. It is not a subset of `clue`, `cmmlu`, or `ceval`. The authors compare modern-Chinese scores on AGIEval and CMMLU with these ancient-Chinese scores to argue that contemporary Chinese skill does not transfer. lm_eval's group name `aclue` is the runnable id; it is unrelated to CLUE.\n\n## Saturation and contamination\n\nThe published 2023 table is close to chance (best 37.34%). Whether current Chinese-capable models have since saturated the set was not established from sources opened here. Contamination risk is high: items and keys are public, and several tasks are generated from public classical corpora. Treat a strong 2026 score as possibly mixed with training-data overlap unless the reporter used a private hold-out, which ACLUE does not provide.\n\n## How to run it\n\nIn lm-evaluation-harness, `aclue` runs all 15 subjects; `aclue_` runs one (for example `aclue_ancient_chinese_culture`). Prompts are Chinese and ask for the option letter after `\u7b54\u6848\uff1a`. The original repository's `src` and `script` directories hold the authors' own model runners. Do not compare a size-weighted lm_eval mean with an unweighted paper mean without saying so.\n\n## Reading the numbers\n\nA score near 25% means the model is not reading classical Chinese above guessing. A 2023-era 37% meant a small edge, not mastery. A much higher modern score would show some classical-Chinese competence, but not that the model can translate, punctuate, or historically reason outside these 15 templates. Read it beside a modern-Chinese suite (`cmmlu`, `clue`) and do not fold it into a CLUE composite.\n", "build": { "built_at": "2026-09-09T16:56:50+00:00", "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11", "eligibility_as_of": "2026-09-09" }, "disposition": { "canonical_id": "aclue", "reasons": [], "status": "unassessed", "verified_results": [] }, "models_covered": [], "page": { "aliases": [ "Ancient Chinese Language Understanding Evaluation" ], "category": "knowledge", "contamination": { "note": "Development and test items, including answers, are public on GitHub and Hugging Face. Eight tasks are generated from public classical corpora. No contamination study was found. The HF card YAML names CC-BY-NC-4.0 while the card body and GitHub README name CC BY-NC-SA 4.0 for the dataset; this page follows the ShareAlike text.\n", "risk": "high" }, "dataset": { "languages": [ "zh" ], "license": "CC-BY-NC-SA-4.0", "modalities": [ "text" ], "public_test_set": true, "size": 4967, "size_note": "Sum of the Hugging Face card's per-task \"Total Q.\" column across all 15 tasks (seven generated tasks at 500 items, then 406, 211, 160, 136, 103, 249, 101, 101). The paper says each task has 100 to 500 questions; 8 tasks are generated from corpora, 5 collected from free tests, and 2 sourced from other work (reading comprehension from AGIEval; basic ancient Chinese from CMMLU). The appendix table labels those last seven as \"collected\". Each task also has a 5-question development split used for few-shot prompts. Whether those five rows are inside the 4,967 figure is not stated on the card. The HF YAML size_categories tag \"1M