Korean legal suite: 7 knowledge tasks, 4 reasoning tasks, and Korean bar-exam items, scored as letter exact match in lm-eval.
unassessed
| Category | domain |
|---|---|
| Subcategory | Korean legal knowledge, legal reasoning, and Korean bar-exam multiple choice |
| Page status | active |
| Metric | exact_match on the extracted option letter (ignore_case; punctuation handling differs by block) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 3456 |
| Dataset licence | CC-BY-NC-4.0 (Hugging Face card); paper also states CC BY-NC for code/data release. Korean bar-exam source texts are KOGL Type 1 per the paper. |
| Publisher | LBox |
KBL tests Korean legal language understanding in three blocks. Seven knowledge tasks (510 examples) cover legal concepts, offence elements, statute matching, statute-number matching, statute hallucination, and common legal mistakes (with and without rationales). Four reasoning tasks (288 examples) cover causal responsibility, statement consistency, and whether a precedent is relevant to a query or to another precedent. The third block is Korean bar-exam multiple-choice items across civil, criminal, public law, and professional responsibility. Knowledge and reasoning items were built with lawyers for this benchmark. The paper also defines a RAG setting over Korean statutes and precedents; EleutherAI lm-eval's in-tree tasks are the closed-book exact-match letter tasks, not that full RAG loop.
Zero-shot Korean generation (output_type generate_until). Prompts present lettered options and ask for a short "답변: A" style answer. Knowledge items use A-C or A-E depending on the task; bar-exam yaml lists A-E. A regex filter takes the first [A-E] from the completion. Gold is the label/gt field. The paper's RAG setting is a separate custom harness branch (lbox-kr/lm-evaluation-harness-kbl), not the in-tree tasks.
No model card in ModelSpec reports this benchmark yet.