Tests applying an invented, never-seen rule across five categories; despite the id, this benchmark (KOR-Bench, "Knowledge-Orthogonal Reasoning") is English-language, not Korean.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | knowledge-orthogonal reasoning across five invented-rule categories (operation, logic, cipher, puzzle, counterfactual) |
| Page status | active |
| Metric | accuracy (exact match after regex/SymPy answer extraction), per category and averaged into an overall score |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1250 |
| Dataset licence | Apache-2.0, per the GitHub repository's LICENSE file (confirmed via the GitHub API); the paper states only that the dataset would be made public on publication, without a separate licence statement for the data itself. |
| Publisher | Multi-institution collaboration (including Tongji University, University of Illinois Urbana-Champaign, Carnegie Mellon University and Nanjing University) with industry labs ByteDance, 01.AI and 2077.AI |
Despite its id and the surface resemblance to this repository's Korean-language benchmarks (csatqa, kormedmcqa), korbench is not Korean. Opening the OpenCompass source the census hint named shows the id refers to KOR-Bench, where "KOR" stands for "Knowledge-Orthogonal Reasoning": an English-language benchmark testing whether a model can apply a freshly defined, out-of-distribution rule to answer a question about it, rather than lean on prior domain knowledge from pretraining. It covers five categories: Operation (novel arithmetic or logical operators), Logic (deductive and inductive puzzles under an invented rule system), Cipher (decoding text under an invented substitution or transformation scheme), Puzzle (constraint-satisfaction and pattern problems) and Counterfactual (reasoning under a stated hypothetical that contradicts real-world facts). Each category is built from 25 invented rules with 10 questions per rule, for 1,250 core questions. Every prompt includes the rule's full definition, so a model with strong general reasoning but no prior exposure to that specific rule should still be able to answer correctly.
A natural-language description of one invented rule plus a question that requires applying it; the model answers in free text, which is extracted by regular-expression parsing and graded by a category-specific script (SymPy handles parsing and comparison of mathematical expressions). Chat models are evaluated zero-shot; base models are evaluated three-shot with worked examples of the same rule type. A separate, harder "Mixed" setting (Multi-Q, Multi-R, Multi-RQ) combines multiple questions or multiple rules in one prompt.
No model card in ModelSpec reports this benchmark yet.