Tests Chinese-specific commonsense reasoning against a matched globally-known-commonsense control, plus free-form memorization questions built from the same underlying facts.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | Chinese commonsense reasoning and knowledge memorization |
| Page status | active |
| Metric | accuracy (multiple-choice for reasoning tasks; LLM-judged or rule-matched for memorization tasks) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 2559 |
| Dataset licence | Apache-2.0 |
| Publisher | Shanghai AI Laboratory; Tongji University; Wuhan University |
CHARM evaluates a model's commonsense reasoning in Chinese across two matched domains: globally known commonsense and Chinese-specific commonsense, the latter spanning seven aspects (history, traditional culture and arts, daily life and customs, entertainment, public figures, geography, and the Chinese language itself). The same seven task types -- such as judging historical anachronisms, understanding a time expression, or recommending a movie -- are built in both domains with identical formats, so a model's gap between the two domains isolates how much of its performance depends on Chinese-specific knowledge rather than general reasoning ability. A separate set of memorization tasks, built from the same underlying facts as four of the reasoning tasks, tests whether a model actually knows the relevant facts, to distinguish reasoning failures from simple forgetting.
14 multiple-choice reasoning subtasks (7 task types, each run in the Chinese-specific and global domains), with 2 to 6 options depending on the subtask, plus 4 free-form question-answering memorization subtasks derived from 4 of the 7 reasoning task types. The original paper evaluated 19 LLMs under 5 prompt strategies, including chain-of-thought and a cross-lingual-thought (XLT) strategy; the public leaderboard reports each model under whichever strategy (XLT for English-oriented models, Chinese chain-of-thought for Chinese-oriented models) scored best for it.
No model card in ModelSpec reports this benchmark yet.