LogiQA scores multiple-choice logical reasoning questions taken from China's National Civil Servants Examination, translated into English by professional translators.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | logical reasoning reading comprehension, from Chinese civil-service exam questions |
| Page status | active |
| Metric | accuracy (and length-normalized accuracy, acc_norm) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 8678 |
| Dataset licence | Not stated by the paper or by the original GitHub repository, which carries no LICENSE file; the authors describe the dataset only as "freely available." The EleutherAI/logiqa Hugging Face mirror used by lm-evaluation-harness lists its own licence tag as "other" without further detail. |
| Publisher | Fudan University and Westlake University |
LogiQA tests deductive logical reasoning through reading comprehension: the model is given a short passage and a question, then must pick the correct one of four candidate answers. Questions come from "publically available questions of the National Civil Servants Examination of China," a real exam designed to test critical thinking rather than recall, and were not written for the benchmark. The paper organises items into five overlapping categories of deductive reasoning -- categorical, sufficient conditional, necessary conditional, disjunctive and conjunctive reasoning -- though a single question can draw on more than one category at once.
Four-option multiple-choice reading comprehension: a short context passage, a question about it, and four answer candidates, with the model returning a single choice. The dataset was released in both its original Chinese and a professionally produced English translation (five translators, three proofreaders), so the same questions can be evaluated in either language.
No model card in ModelSpec reports this benchmark yet.