The Chinese counterpart to OpenAI's SimpleQA -- 3,000 short fact-seeking questions across 6 topics and 99 subtopics, from Alibaba's Taobao & Tmall Group, LLM-graded following SimpleQA's approach.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | short-answer factuality, LLM-graded (Chinese-language counterpart to OpenAI's SimpleQA) |
| Page status | active |
| Metric | LLM-grader-classified correctness, following SimpleQA's grading approach |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 3000 |
| Dataset licence | CC-BY-NC-SA-4.0, per the Hugging Face dataset card; the OpenStellarTeam/ChineseSimpleQA GitHub repository carries no separate LICENSE file for its code. |
| Publisher | Taobao & Tmall Group, Alibaba |
Chinese SimpleQA measures short-form factuality in Chinese: whether a model gives a correct, brief answer to a fact-seeking question with one static, indisputable answer. The paper positions the benchmark explicitly as a Chinese counterpart to OpenAI's SimpleQA (documented in this repository as `simpleqa`), built to address the same gap OpenAI identified -- that short, open-ended fact-seeking evaluation was missing -- but for Chinese, where the authors say the widely-used knowledge sets (CommonSenseQA, CMMLU, C-Eval) are multiple-choice rather than open-ended. Questions span six major topics -- Chinese Culture, Humanities, Engineering/ Technology/Applied Sciences, Life/Art/Culture, Society, and Natural Science -- covering 99 finer-grained subtopics, and the authors state they applied "a comprehensive and rigorous quality control process" to keep reference answers static (unchanging over time) and unambiguous. It is a single-turn, Chinese-language, text-only task, independently constructed for Chinese rather than translated from OpenAI's English question set.
Short free-form answer generation to a single fact-seeking Chinese-language question, with no supporting passage supplied; graded by a separate LLM grader (the repository recommends the OpenAI API) against a reference answer, following SimpleQA's grading approach, rather than scored by exact string match.
No model card in ModelSpec reports this benchmark yet.