{
 "body": "\n## What it measures\n\nHRM8K evaluates multilingual mathematical reasoning. The Korean benchmark contains 8,011 problems with Korean instruction and questions; the harness also documents an English `hrm8k_en` version.\n\nThe collection combines translations from GSM8K, MATH, OmniMath, and MMMLU with original problems curated from Korean mathematics exams. It is intended to study the gap between solving in English and understanding Korean input.\n\n## How it is scored\n\nThe harness task family includes GSM8K, KSM, MATH, MMMLU, and OmniMath variants in Korean and English. It recommends temperature 0.7, top-p 0.95, and up to 2,048 generated tokens, while the default harness decoding is greedy. The README does not give one aggregate formula or human baseline.\n\n## Dataset and licence\n\nThe official harness README states 8,011 evaluation instances and links `HAERAE-HUB/HRM8K`. The benchmark, training data, and models are publicly released. The Hugging Face dataset card lists an MIT licence, and each of the five source-task configs ships a single `test` split (for example `gsm8k_test.csv`).\n\n## Who publishes it\n\nKo, Son, and Choi introduced HRM8K in \u201cUnderstand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap,\u201d arXiv:2501.02448. EleutherAI\u2019s harness integrates the tasks. No current leaderboard was established.\n\n## Lineage\n\nHRM8K has Korean and English variants and five named source-task variants. They should be reported separately because language and source benchmark differ.\n\n## Saturation and contamination\n\nThe benchmark remains open in the available evidence. Public source datasets create medium exposure risk, and synthetic training data are also public. Exposure may differ between translated Korean items and original exam problems.\n\n## How to run it\n\nUse the `hrm8k` task group or a specific `hrm8k_{gsm8k|ksm|math|mmmlu|omni_math}` task. The English counterparts use the `hrm8k_en_` prefix. Record decoding parameters, task variant, and harness version.\n\n## Reading the numbers\n\nA high score indicates mathematical problem-solving success in a specified language and source family. It does not isolate translation quality from mathematical reasoning, and it does not establish broad Korean language competence. Compare language-matched variants and decoding settings.\n\nThe stated research goal distinguishes comprehension from mathematical reasoning. That interpretation requires paired Korean and English results on comparable items, rather than one pooled score.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "hrm8k",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "math",
  "contamination": {
   "and models are public; exposure may vary by language and source.": null,
   "note": "The benchmark",
   "risk": "medium",
   "training data": null
  },
  "dataset": {
   "languages": [
    "Korean",
    "English"
   ],
   "license": "MIT",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 8011,
   "size_note": "The harness README describes 8,011 evaluation instances.",
   "splits": "test",
   "url": "https://huggingface.co/datasets/HAERAE-HUB/HRM8K"
  },
  "freshness": {
   "luna-batch-048": null,
   "luna-batch-048 (Codex coordinated)": null,
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "hrm8k",
   "opencompass": "",
   "other": ""
  },
  "id": "hrm8k",
  "last_updated": "2025-07",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "hrm8k_en"
   ]
  },
  "measures": "HRM8K tests mathematical problem solving when instructions and questions are presented in Korean, with aligned English versions. It combines translated problems from established benchmarks with original Korean exam problems.",
  "metric": {
   "baseline_note": "The harness README does not define a human baseline.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": null,
   "unit": "percent"
  },
  "name": "HRM8K",
  "page_kind": "family",
  "paper": {
   "arxiv": "2501.02448",
   "title": "Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap",
   "url": "https://arxiv.org/abs/2501.02448",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Hyunwoo Ko",
    "Guijin Son",
    "Dasol Choi"
   ],
   "org": "HAERAE-HUB",
   "url": "https://huggingface.co/datasets/HAERAE-HUB/HRM8K"
  },
  "released": "2025",
  "repo_url": "https://huggingface.co/datasets/HAERAE-HUB/HRM8K",
  "saturation": {
   "as_of": "",
   "note": "No current authoritative leaderboard was established.",
   "status": "open",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness HRM8K README",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/hrm8k/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "HRM8K paper",
    "url": "https://arxiv.org/abs/2501.02448"
   },
   {
    "accessed": "2026-09-08",
    "title": "HRM8K dataset card",
    "url": "https://huggingface.co/datasets/HAERAE-HUB/HRM8K"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face dataset API record (license mit, five test-split configs)",
    "url": "https://huggingface.co/api/datasets/HAERAE-HUB/HRM8K"
   }
  ],
  "status": "active",
  "subcategory": "multilingual mathematical reasoning",
  "summary": "HRM8K evaluates Korean and English mathematical reasoning on 8,011 parallel bilingual problems.",
  "tags": [
   "math",
   "korean",
   "multilingual"
  ],
  "task_format": "Free-form mathematical solution generation in Korean or English."
 }
}