An OpenCompass-maintained, Chinese-language STEM QA benchmark split into biology, chemistry and physics subsets, graded like OpenAI's SimpleQA; its question set is not publicly downloadable.
unassessed
| Category | domain |
|---|---|
| Subcategory | free-response biology, chemistry and physics QA, periodically re-versioned, SimpleQA-style grading |
| Page status | active |
| Metric | accuracy_given_attempted (precision on attempted answers) and F1, following SimpleQA's own metric design |
| Direction | higher_is_better |
| Unit | % |
| Publisher | Shanghai Artificial Intelligence Laboratory (OpenCompass project) |
LiveStemBench poses a single Chinese-language free-response question in one of three subjects -- biology, chemistry or physics -- and compares the model's answer against a gold target. Its dataset loader is explicitly commented in the source code as "Edited from the official SimpleQA config," and its grading prompt reuses OpenAI's SimpleQA rubric and worked examples verbatim. Some items carry multiple-choice-style options that the loader folds directly into the question text before presenting it to the model, but the model is still expected to produce a free-text answer compared against a gold target, not a lettered choice. As with this batch's `livereasonbench`, the exact topic coverage and difficulty could not be independently confirmed for this page because the dataset is not publicly downloadable; what is confirmed comes from the harness source code itself.
Single-turn free-response question answering in Chinese, with three documented prompting variants in the harness: a chain-of-thought variant asking the model to "think step by step" and box its final answer, a direct-answer ("0shot_noncot") variant with no reasoning instruction, and a variant of the latter that also requires the answer inside an XML `<conclude>` tag for extraction. All three are graded by a separate LLM-judge call against a gold target.
No model card in ModelSpec reports this benchmark yet.