{
 "body": "\n## What it measures\n\nOJBench measures competitive programming ability at the level of NOI and ICPC problems. It targets code reasoning and solution generation rather than short code completion.\n\nThe paper describes 232 problems and evaluates a range of closed and open models. A model receives a programming problem and must generate a solution that passes the benchmark\u2019s judging procedure.\n\n## How it is scored\n\nThe paper reports Pass@8: eight candidate solutions are generated and a problem counts as passed when the judging procedure accepts a solution. The paper describes complete test-case judging and supports Python and C++; exact compiler and timeout settings remain implementation details.\n\n## Dataset and licence\n\nThe paper reports 232 problems from NOI and ICPC, with 159 from NOI and 73 from ICPC. The public OpenCompass path is `opencompass/ojbench`. A dataset licence and split structure were not established.\n\n## Who publishes it\n\nOJBench was introduced in the 2025 paper by Wang and colleagues (arXiv:2506.16395). OpenCompass publishes a runnable integration and dataset path. No current standalone leaderboard was established.\n\n## Lineage\n\nOJBench is a standalone competitive-programming benchmark. The sources read do not establish a predecessor, successor, or repository variant.\n\n## Saturation and contamination\n\nThe paper reports that even leading reasoning models struggle, supporting an open status. The underlying contest problems are publicly available, so training exposure is plausible. No dedicated contamination study was established.\n\n## How to run it\n\nUse OpenCompass\u2019s `ojbench` configuration with zero-shot retrieval and generation. It prompts the model with the problem text and evaluates generated code using an online judge. Record the language, execution limits, judge version, and number of candidates because they materially affect Pass@8.\n\n## Reading the numbers\n\nA high pass rate indicates that generated programs satisfy the judge on this difficult contest set. It does not measure maintainability, explanation quality, or software engineering outside contest constraints. Compare only runs with the same language, tool access, time limits, and judge implementation.\n\nThe benchmark\u2019s difficulty also makes failure analysis valuable. A pass rate alone does not distinguish parsing mistakes, algorithmic errors, resource-limit failures, and incomplete reasoning, so reports should retain per-problem outcomes where possible.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "ojbench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "coding",
  "contamination": {
   "but the paper\u2019s exposure analysis was not established from the source read.": null,
   "note": "Competition problems are public",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "license": "",
   "modalities": [
    "text",
    "code"
   ],
   "public_test_set": true,
   "size": 232,
   "size_note": "159 NOI and 73 ICPC problems; difficulty counts are 36 easy, 79 medium, and 117 hard.",
   "splits": "",
   "url": "https://huggingface.co/datasets/opencompass/ojbench"
  },
  "freshness": {
   "luna-batch-017": null,
   "luna-batch-017 (Codex coordinated)": null,
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna",
   "reviewed": "2026-09-08",
   "reviewed_by": "GPT-5.6 Luna independent review"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "ojbench",
   "other": "online judge"
  },
  "id": "ojbench",
  "last_updated": "2026-03",
  "leaderboard_url": "https://he-ren.github.io/OJBench/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "OJBench tests whether a model can reason about and produce solutions for difficult programming-contest problems. The paper describes problems drawn from NOI and ICPC competitions.",
  "metric": {
   "baseline_note": "The paper evaluates eight candidate solutions per problem.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "Pass@8",
   "random_baseline": null,
   "unit": "percent"
  },
  "name": "OJBench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2506.16395",
   "title": "OJBench: A Competition Level Code Benchmark For Large Language Models",
   "url": "https://arxiv.org/abs/2506.16395",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Zhexu Wang",
    "Yiping Liu",
    "Yejie Wang",
    "Wenyang He",
    "Bofei Gao",
    "Muxi Diao",
    "Yanxu Chen",
    "Kelin Fu",
    "Flood Sung",
    "Zhilin Yang",
    "Tianyu Liu",
    "Weiran Xu"
   ],
   "org": "OJBench authors",
   "url": "https://arxiv.org/abs/2506.16395"
  },
  "released": "2025-06",
  "repo_url": "https://github.com/open-compass/opencompass",
  "saturation": {
   "as_of": "",
   "note": "The paper reports that state-of-the-art reasoning models still struggle; no current top score was established.",
   "status": "open",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "OJBench paper",
    "url": "https://arxiv.org/abs/2506.16395"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass OJBench configuration",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/ojbench/ojbench_gen.py"
   }
  ],
  "status": "active",
  "subcategory": "competitive programming",
  "summary": "OJBench evaluates competitive-level code reasoning on 232 NOI and ICPC programming problems.",
  "tags": [
   "code-generation",
   "competitive-programming",
   "reasoning"
  ],
  "task_format": "Natural-language programming problem prompt; generate a code solution evaluated by an online-judge style checker."
 }
}