Expanded civil-service logical-reasoning suite: English and Chinese four-option MRC, plus a two-way NLI conversion of the same items.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | civil-service logical-reasoning MRC (English/Chinese) plus two-way NLI |
| Page status | active |
| Metric | accuracy (lm-eval also reports acc_norm on the English MRC task) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 15708 |
| Dataset licence | CC BY-NC-SA 4.0 (GitHub README and the baber/logiqa2 loader's _LICENSE string). The Hugging Face cardData license field is cc-by-sa-4.0, which drops NonCommercial; treat the repository statement as the dataset licence. |
| Publisher | Westlake University and collaborators; translation funded by Microsoft Research Asia and annotated by Speechocean |
LogiQA 2.0 reworks [LogiQA](logiqa.md). Items still come from China's civil-service examination materials, now with more questions, professional re-translation, and removal of culturally specific wording such as Chinese idioms. The main protocol is four-option reading comprehension: a passage, a question, and four answers. A second protocol converts each MRC item into premise-hypothesis pairs labelled entailed or not entailed. lm-evaluation-harness's `logiqa2` task scores the English MRC split only.
MRC: four-option multiple choice; gold `answer` is an integer 0-3. NLI: two-way label entailed / not entailed over major_premise, minor_premise, and conclusion. English and Chinese MRC files are released separately. lm-eval `logiqa2` is zero-shot multiple choice with acc and acc_norm. A second harness task, `logieval`, is a 1-shot generate-until prompt from csitfun/LogiEval, not the paper's MRC loader.
No model card in ModelSpec reports this benchmark yet.