OpenCompass multiple-choice set (abbrs kcle and kcle_fix) graded by an LLM judge; item count, licence and the acronym's expansion are not established.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | OpenCompass local multiple-choice set graded by an LLM judge (expansion of KCLE/KLCE not established) |
| Page status | unknown |
| Metric | LLM-judge accuracy (GenericLLMEvaluator, A=CORRECT / B=INCORRECT) |
| Direction | higher_is_better |
| Unit | % |
| Publisher | OpenCompass / Shanghai AI Laboratory (harness packaging); original dataset authors not named in the loader |
kcle is OpenCompass's KCLEDataset: a local jsonl of input/target pairs that the default configs treat as multiple-choice questions and score with an LLM judge. The mapping table comments the block as "KLCE Datasets" while paths and class names use kcle, so the acronym itself is not established from a paper. Local filenames are kcle_diamond.jsonl and kcle_diamond_fix_251029.jsonl. That "diamond" token is not evidence that this id is [gpqa_diamond](gpqa_diamond.md); GPQA Diamond is a separate 198-item science set with its own page. What the items actually ask is not described beyond the generic multiple-choice prompt, because the jsonl files and Hub paths were not readable here.
Zero-shot generation (ZeroRetriever + GenInferencer). One config feeds the raw input field. The kcle_fix configs prepend an English instruction to end with ANSWER: $LETTER. A second model (GenericLLMEvaluator) then grades the completion against the gold target as A/CORRECT or B/INCORRECT.
No model card in ModelSpec reports this benchmark yet.