{
 "body": "\n## What it measures\n\nKoBEST asks whether a model can do five Korean NLU jobs that linguists wrote by hand.\nBoolQ is a paragraph plus a yes/no question. COPA is a premise plus a cause/effect\nchoice between two alternatives. WiC asks if one word means the same thing in two\nsentences. HellaSwag continues a short scene with four endings. SentiNeg labels a\nproduct-review sentence as positive or negative, with negation as the hard part.\nThe language is Korean text only. This is not [korbench](korbench.md).\n\n## How it is scored\n\nThe COLING paper reports test F1 (Table 4). Fine-tunes were run five times and\nshown as mean \u00b1 std. Human F1 is 10 native speakers on 100 random items per task,\nnot the full test files. lm-evaluation-harness scores each task as multiple choice.\nEvery YAML lists accuracy and sklearn macro F1. HellaSwag also lists length-normalised\naccuracy. The group `kobest` averages acc, acc_norm, and F1 weighted by dataset\nsize. Do not mix a paper F1 from fine-tuning with a harness zero-shot accuracy.\n\n## Dataset and licence\n\nHugging Face `skt/kobest_v1` is the copy the harness loads. The card and API mark\nthe licence CC-BY-SA-4.0. The README \u201cLicensing Information\u201d block is only a\ncitation, not a second licence. datasets-server test counts are 1,404 / 1,000 /\n500 / 397 / 1,260 for BoolQ, COPA, HellaSwag, SentiNeg, and WiC. Paper Table 1\nmatches BoolQ, HellaSwag, and SentiNeg, but lists larger COPA and WiC development\nsplits than the Hub files. The README repeats BoolQ\u2019s counts for HellaSwag; the\nJSONL files do not. SentiNeg also ships `test_originated`, the reviews the test\nitems were flipped from. Labels are public.\n\n## Who publishes it\n\nSK Telecom\u2019s Language Super Intelligence Labs released the data. Myeongjun Jang\nis listed with Oxford on the arXiv HTML. COLING 2022 (October, Gyeongju) is the\npeer-reviewed paper; authors there are Jang, Dohyung Kim, Deuk Sin Kwon, and\nEric Davis. The arXiv v1 listing puts Dohyeong Kim first and spells the given\nname differently. Hugging Face created the dataset on 2022-04-07. There is no\nlive official leaderboard; Table 4 in the paper is the published score table.\n\n## Lineage\n\nThe five tasks follow English BoolQ, COPA, WiC, HellaSwag, and a negation-focused\nsentiment setup, with new Korean items rather than translations. Nothing in this\nrepository is a subset page. Do not treat [korbench](korbench.md) as a Korean\nsibling. No successor suite is recorded here.\n\n## Saturation and contamination\n\nOn the 2022 table, KoElectra\u2019s 80.6 mean F1 still sits well below the 96.2 human\nsample. How current generative models score on the lm-eval group was not read\nhere, so saturation is unknown. Every labelled split has been downloadable since\nApril 2022, so leakage into later pretraining is plausible.\n\n## How to run it\n\nIn lm-evaluation-harness, run the group `kobest` or a single `kobest_*` task.\nThe README\u2019s `kobest_hallawag` name is a typo; the YAML task is\n`kobest_hellaswag`. Prompts are Korean strings in the YAML and `utils.py`.\nBoolQ and WiC use \uc544\ub2c8\uc624/\uc608. SentiNeg uses \ubd80\uc815/\uae0d\uc815. COPA inserts \uc65c\ub0d0\ud558\uba74 or\n\uadf8\ub798\uc11c from the cause/effect field. Compare only runs that share shot count\nand whether they use acc or F1.\n\n## Reading the numbers\n\nA strong group score means the model handles Korean QA, causal choice, sense,\ncompletion, and negated polarity together. It does not mean Korean legal or\nexam knowledge. Paper F1 from fine-tuned encoders is not the same protocol as\nlm-eval loglikelihood multiple choice. Check the Hub split you loaded: COPA\nand WiC validation counts disagree between Table 1 and the files. Report the\nfive task scores, not only the size-weighted mean.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "kobest",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "KOBEST",
   "KB-BoolQ",
   "KB-COPA",
   "KB-WiC",
   "KB-HellaSwag",
   "KB-SentiNeg"
  ],
  "category": "composite",
  "contamination": {
   "note": "All splits including labels have been public on Hugging Face since the dataset was created 2022-04-07 (card lastModified 2024-03-28). Items were written for the benchmark, not scraped from a hidden exam.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "ko"
   ],
   "license": "CC-BY-SA-4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 4561,
   "size_note": "Hugging Face datasets-server (skt/kobest_v1) test rows: BoolQ 1,404; COPA 1,000; HellaSwag 500; SentiNeg 397; WiC 1,260 (4,561 scored if the group runs all five tests). Train/validation on the same API: BoolQ 3,665/700; COPA 3,076/500; HellaSwag 2,029/500; SentiNeg 3,649/400 plus a test_originated split of 397; WiC 3,318/610. Paper Table 1 and the HF README list COPA dev 1,000 and WiC dev 1,260; those files on the Hub are 500 and 610. The README also copies BoolQ counts onto HellaSwag; the paper and the API both have HellaSwag 2,029/500/500.\n",
   "splits": "per task train/validation/test on Hugging Face; SentiNeg also has test_originated",
   "url": "https://huggingface.co/datasets/skt/kobest_v1"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-053 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-053"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "kobest",
   "opencompass": "",
   "other": "Runnable tasks kobest_boolq, kobest_copa, kobest_hellaswag, kobest_sentineg, kobest_wic. The task README lists kobest_hallawag; that string is not a YAML task name."
  },
  "id": "kobest",
  "last_updated": "2024-03",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "KoBEST is a Korean-only text suite of five multiple-choice NLU tasks. BoolQ asks whether a question is true given a paragraph. COPA picks which of two Korean alternatives is the cause or effect of a premise. WiC asks whether a word has the same sense in two sentences. HellaSwag picks the plausible next sentence from four endings. SentiNeg labels a review sentence as positive or negative, with items built around negation. Professional linguists designed the items. Not [korbench](korbench.md).\n",
  "metric": {
   "baseline_note": "Paper Table 4 reports test F1. Human row 95.1 / 98.1 / 96.6 / 92.4 / 99.0 (mean 96.2) is 10 native Korean raters on 100 random items per task, not the full test split. Best fine-tune in that table: KoElectra 80.6 mean F1. Chance is not one number (2-way vs 4-way tasks). lm-eval group aggregates size-weighted mean acc, acc_norm, and F1.\n",
   "direction": "higher_is_better",
   "human_baseline": 96.2,
   "max_score": 100,
   "name": "macro F1 (paper); lm-eval also reports accuracy (and acc_norm on HellaSwag)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "KoBEST (Korean Balanced Evaluation of Significant Tasks)",
  "page_kind": "family",
  "paper": {
   "arxiv": "2204.04541",
   "title": "KoBEST: Korean Balanced Evaluation of Significant Tasks",
   "url": "https://aclanthology.org/2022.coling-1.325/",
   "year": 2022
  },
  "publisher": {
   "authors": [
    "Myeongjun Jang",
    "Dohyung Kim",
    "Deuk Sin Kwon",
    "Eric Davis"
   ],
   "org": "SK Telecom Language Super Intelligence Labs; University of Oxford (Jang)",
   "url": "https://huggingface.co/datasets/skt/kobest_v1"
  },
  "released": "2022-04",
  "repo_url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/kobest",
  "saturation": {
   "as_of": "2022-10",
   "note": "Paper Table 4 best mean test F1 among fine-tuned Korean LMs is KoElectra 80.6, with humans at 96.2 on 100-item samples. No dated generative-LLM group score was opened here.\n",
   "status": "unknown",
   "top_score": 80.6
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness kobest README (group, task list, paper link)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/kobest/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "_kobest.yaml (group kobest; size-weighted acc, acc_norm, f1)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/kobest/_kobest.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "kobest_boolq.yaml (skt/kobest_v1 boolq; \uc544\ub2c8\uc624/\uc608; acc and macro F1)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/kobest/kobest_boolq.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "kobest_copa.yaml",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/kobest/kobest_copa.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "kobest_hellaswag.yaml (acc, acc_norm, F1)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/kobest/kobest_hellaswag.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "kobest_sentineg.yaml",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/kobest/kobest_sentineg.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "kobest_wic.yaml",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/kobest/kobest_wic.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "kobest utils.py (Korean prompts and macro_f1_score)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/kobest/utils.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "skt/kobest_v1 dataset card (CC-BY-SA-4.0; split tables; task descriptions)",
    "url": "https://huggingface.co/datasets/skt/kobest_v1"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face dataset API (created 2022-04-07; lastModified 2024-03-28; license cc-by-sa-4.0)",
    "url": "https://huggingface.co/api/datasets/skt/kobest_v1"
   },
   {
    "accessed": "2026-09-08",
    "title": "datasets-server split counts for all five configs",
    "url": "https://datasets-server.huggingface.co/info?dataset=skt/kobest_v1"
   },
   {
    "accessed": "2026-09-08",
    "title": "KoBEST paper HTML (Table 1 sizes; Table 4 F1 including human 96.2)",
    "url": "https://ar5iv.labs.arxiv.org/html/2204.04541"
   },
   {
    "accessed": "2026-09-08",
    "title": "arXiv abs 2204.04541 (submitted 2022-04-09; authors Kim, Jang, Kwon, Davis)",
    "url": "https://arxiv.org/abs/2204.04541"
   },
   {
    "accessed": "2026-09-08",
    "title": "COLING 2022 anthology page (October 2022; Jang, Dohyung Kim, Kwon, Davis)",
    "url": "https://aclanthology.org/2022.coling-1.325/"
   }
  ],
  "status": "active",
  "subcategory": "five-task Korean NLU suite (BoolQ, COPA, WiC, HellaSwag, SentiNeg)",
  "summary": "Five human-written Korean NLU tasks (yes/no QA, causal alternatives, word sense, sentence completion, polarity under negation) scored as multiple-choice accuracy and macro F1.",
  "tags": [
   "korean",
   "nlu",
   "multiple-choice",
   "lm-eval",
   "glue-style"
  ],
  "task_format": "lm-eval multiple_choice on skt/kobest_v1. Group name kobest. Runnable tasks kobest_boolq, kobest_copa, kobest_hellaswag, kobest_sentineg, kobest_wic. Korean prompts; yes/no choices \uc544\ub2c8\uc624/\uc608 on BoolQ and WiC; \ubd80\uc815/\uae0d\uc815 on SentiNeg.\n"
 }
}