{
 "body": "\n## What it measures\n\nCLIcK asks whether a model knows Korean language and Korean cultural context well enough to pass exam-style questions. Each item is a Korean stem, optional paragraph, and lettered choices. Categories split into language (textual, grammar, function) and culture (society, tradition, politics, economy, law, history, geography, K-pop).\n\nThe authors built the set from two streams. They reclassified official exam items (CSAT, TOPIK, PSAT, Kedu, PSE, KHB). They also prompted GPT-4 on KIIP textbook chapters and kept validated questions. The skill is Korean-centric knowledge, including items that a translated MMLU clone would miss. It is not [csatqa](csatqa.md), which is a smaller HAE-RAE CSAT slice with human student rates.\n\n## How it is scored\n\nlm-eval reports mean `acc` and `acc_norm`, weighted by subset size, on the group `click`. Subgroups `click_lang` and `click_cul` average their tagged tasks the same way. The target is the letter whose choice string matches the dataset answer. When the example id contains `CSAT`, `get_choices` and `get_target` use A\u2013E; otherwise they use A\u2013D. The Korean `get_context` prompt still always prints A\u2013D, so a fifth CSAT option is scored but not shown in that stem.\n\nThe paper evaluated 13 models. API models were scored by generated text; open models by token likelihood. Claude-2 led at 51.72% culture and 45.39% language. That is the published reference table, not an lm-eval leaderboard. There is no single random baseline for the mixed 4- and 5-choice items.\n\n## Dataset and licence\n\nThe paper's Table 1 and Hugging Face datasets-server both give 1,995 examples. Culture sums to 1,345 and language to 650. 1,245 items are listed as textbook-origin and 750 as exam-origin. The Hub config exposes a single `train` split; lm-eval evaluates that split. Answers are public.\n\nNo licence file was present in the GitHub tree opened here, and the Hub card does not set a licence field. Leave licence empty rather than infer CC-BY. The Hub card still contains unused `your_username/CLIcK` badge links; the live dataset id is `EunsuKim/CLIcK`.\n\n## Who publishes it\n\nEunsu Kim, Haneul Yoo, and Alice Oh (KAIST School of Computing) with Juyoung Suk, Philhoon Oh, and James Thorne (KAIST GSAI) released the dataset and the LREC-COLING 2024 paper (arXiv 2403.06412, 11 March 2024). The GitHub repo is `rladmstn1714/CLIcK`. Hugging Face lastModified on the API blob opened here is 7 September 2024. EleutherAI maintains the lm-eval task group.\n\n## Lineage\n\nCLIcK was built because translated English culture tests miss Korean context, and existing Korean culture sets were narrow (bias or hate speech). Exam overlap with [csatqa](csatqa.md) is possible on CSAT Korean items, but csatqa is a different 187-question harness group with human scores. [kormedmcqa](kormedmcqa.md) is medical. [korbench](korbench.md) is not Korean.\n\nlm-eval filters categories in `utils.py` (for example CSAT Korean id ranges for text vs grammar). Those filters implement the eleven-way split; they are not extra hidden tests.\n\n## Saturation and contamination\n\nClaude-2 at about 52% culture in 2024 left clear headroom. No later official table was found in this research, so saturation stays open with that paper figure as a dated reference, not a 2026 ceiling.\n\nExam questions can leak. Textbook-generated items are newer but public. Treat the set as a medium contamination risk.\n\n## How to run it\n\n```bash\nlm-eval --tasks click\n```\n\nUse `click_lang`, `click_cul`, or a single task such as `click_cul_history` for slices. Default YAMLs load `EunsuKim/CLIcK` with `output_type: multiple_choice`. The Korean prompt is hard-coded in `utils.get_context`. Few-shot, if requested, draws from the same `train` split the test uses, so in-context examples can overlap the eval pool unless you control sampling.\n\nPaper likelihood scoring of open models is not guaranteed to match a generative lm-eval run.\n\n## Reading the numbers\n\nA high `click` accuracy means the model often picked the keyed letter on this 1,995-item mix. It does not mean fluency in open-ended Korean, and it does not mean medical or legal competence beyond the culture/law questions in the set. Compare `click_lang` and `click_cul` before quoting one number. Do not stack it with [csatqa](csatqa.md) as if they were the same CSAT dump. Claude-2's 51.72% is a 2024 culture average under the paper's protocol, not an lm-eval result on a 2026 model.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "click",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "CLIcK: Cultural and Linguistic Intelligence in Korean",
   "Cultural and Linguistic Intelligence in Korean"
  ],
  "category": "knowledge",
  "contamination": {
   "note": "Exam items (CSAT, TOPIK, PSAT, and others) may appear in pretraining. 1,245 textbook-derived items were generated with GPT-4 from KIIP chapters, which reduces exact exam overlap but can echo textbook wording. The full 1,995 items are public on Hugging Face.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "ko"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 1995,
   "size_note": "Paper Table 1 and Hugging Face datasets-server agree on 1,995 examples in the default train split. Culture 1,345 (society 309, tradition 222, history 280, law 219, politics 84, economy 59, geography 131, pop culture 41); language 650 (textual 285, functional 133, grammar 232). 1,245 textbook / 750 exam in that table.\n",
   "splits": "Hugging Face default config exposes only train (1,995); lm-eval uses that split for both evaluation and few-shot",
   "url": "https://huggingface.co/datasets/EunsuKim/CLIcK"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-031 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-031"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "click",
   "opencompass": "",
   "other": ""
  },
  "id": "click",
  "last_updated": "2024-09",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "click is EleutherAI lm-evaluation-harness's group for CLIcK (Cultural and Linguistic Intelligence in Korean). The model reads a Korean question, often with a short context, and picks a lettered choice. Items test Korean language (textual, grammatical, functional knowledge) and Korean culture (society, tradition, politics, economy, law, history, geography, popular culture). Sources are official exams plus GPT-4 questions written from the KIIP textbook and then validated. This is a Korean-centric knowledge test, not a translated English quiz and not [korbench](korbench.md) (which is English knowledge-orthogonal reasoning despite the similar id).\n",
  "metric": {
   "baseline_note": "Four-option items have a 25% chance rate and CSAT items a 20% rate; the mixed set has no single chance figure in the paper. Claude-2 was the strongest of 13 models in the paper (51.72% culture, 45.39% language). GPT-3.5 was described as the 11th percentile of Korean test-takers on the relevant exams, not as a human mean on this 1,995-item mix.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy and length-normalized accuracy (acc, acc_norm)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "CLIcK",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2403.06412",
   "title": "CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean",
   "url": "https://arxiv.org/abs/2403.06412",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Eunsu Kim",
    "Juyoung Suk",
    "Philhoon Oh",
    "Haneul Yoo",
    "James Thorne",
    "Alice Oh"
   ],
   "org": "KAIST (School of Computing and GSAI)",
   "url": "https://github.com/rladmstn1714/CLIcK"
  },
  "released": "2024-03",
  "repo_url": "https://github.com/rladmstn1714/CLIcK",
  "saturation": {
   "as_of": "2024-03",
   "note": "Paper Table of 13 models: Claude-2 51.72% average on Korean culture and 45.39% on Korean language. Open Korean LMs clustered near 32\u201336% culture and 22\u201327% language. No later public ceiling was found. 51.72 is culture average, not a single official overall.\n",
   "status": "open",
   "top_score": 51.72
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "CLIcK paper (arXiv 2403.06412)",
    "url": "https://arxiv.org/abs/2403.06412"
   },
   {
    "accessed": "2026-09-08",
    "title": "CLIcK paper HTML",
    "url": "https://ar5iv.labs.arxiv.org/html/2403.06412"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face EunsuKim/CLIcK",
    "url": "https://huggingface.co/datasets/EunsuKim/CLIcK"
   },
   {
    "accessed": "2026-09-08",
    "title": "CLIcK dataset card README",
    "url": "https://huggingface.co/datasets/EunsuKim/CLIcK/raw/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "datasets-server info: 1,995 train rows",
    "url": "https://datasets-server.huggingface.co/info?dataset=EunsuKim/CLIcK"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval click README",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/click/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval click group YAML",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/click/click.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval click_lang default YAML (extensionless include file)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/click/click_lang/_default_click_lang_yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval click_lang utils (prompt, CSAT A-E choices)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/click/click_lang/utils.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "CLIcK GitHub repository",
    "url": "https://github.com/rladmstn1714/CLIcK"
   }
  ],
  "status": "active",
  "subcategory": "Korean cultural and linguistic multiple-choice QA from exams and textbooks",
  "summary": "Korean multiple-choice exam of cultural and linguistic knowledge: 1,995 questions in eleven categories, drawn from official exams and textbooks.",
  "tags": [
   "korean",
   "culture",
   "language",
   "multiple-choice",
   "lm-eval"
  ],
  "task_format": "Multiple choice in Korean. lm-eval output_type is multiple_choice. The Korean prompt in utils.get_context always lists A\u2013D. Scoring helpers get_choices and get_target use A\u2013E when the example id contains CSAT, else A\u2013D. The Hugging Face split used as both test_split and fewshot_split is named train. Groups: click (all 11), click_lang (3), click_cul (8).\n"
 }
}