OpenCompass's own composite: naive-averages six category scores (language, knowledge, reasoning, math, code, agent), each itself an average across many pre-existing and self-built datasets.
unassessed
| Category | composite |
|---|---|
| Subcategory | OpenCompass's own composite evaluation aggregating six category groups -- language, knowledge, reasoning, mathematics, code, and agent/tool-use -- each itself a naive average across several pre-existing and self-built datasets |
| Page status | unknown |
| Metric | top-level 'average' -- a naive average of six category-level naive averages (language, knowledge, reasoning, math, code, agent), each itself averaging several differently-scored constituent datasets (circular-eval accuracy, BLEU, ROUGE, pass@1) -- not one measurement |
| Direction | higher_is_better |
| Unit | % |
| Publisher | OpenCompass (Shanghai AI Laboratory) |
CompassBench v1.1 is not a single skill test; it is OpenCompass's own composite evaluation aggregating six category groups, established directly from its dataset and summarizer configurations in the opencompass GitHub repository: language (intention recognition, sentiment analysis, translation, content criticism, content summarization, and Chinese cultural/semantic understanding, mostly paired English/Chinese tasks), knowledge (a WikiBench-derived Chinese single-choice split across common knowledge, humanities, natural science and social science, plus an English TriviaQA-style cloze set), reasoning (ReasonBench, circular multiple-choice items covering abductive, deductive, inductive and commonsense reasoning in English and Chinese, several machine- translated from BBH, LogiQA and OCNLI), mathematics (MathBench, arithmetic through college-level problems in English and Chinese), code (HumanEval, HumanEval+, MBPP, HumanEval-X, LCBench2023 and TACO, spanning multiple languages and three difficulty tiers), and agent/tool-use (CIBench, a code- interpreter task, and T-Eval/plugin_eval, a tool-planning evaluation). Each category score is itself a naive average across its constituent datasets, and the suite's own top-level "average" is a naive average of the six category scores -- three layers of averaging stand between any single reported number and one underlying task, which is why this page's category field is `composite` rather than any single skill.
Overwhelmingly multiple choice with a few generation tasks: most sub-datasets are graded with OpenCompass's `CircularEvaluator`, which permutes a multiple-choice item's answer options across several passes and only credits the model if every permutation is answered correctly (`perf_circular` or `perf_4` for four-option items), reported alongside plain single-pass accuracy (`acc_origin` / `acc_1`). Free-text tasks use task-appropriate metrics: BLEU for translation (against FLORES-style references), ROUGE for summarization, and functional test execution (pass@1 / pass@k) for the code category's HumanEval-, MBPP- and TACO-derived tasks. Knowledge and math use 4-shot in-context prompting; most other categories are zero-shot.
No model card in ModelSpec reports this benchmark yet.