{
 "body": "\n## What it measures\n\nNPHardEval measures algorithmic reasoning with generated answers to graph coloring, traveling-salesperson, matching, and path problems. The official description says its 900 questions span a broad range of complexity classes up to NP-hard problems.\n\nIt includes optimization-style and decision-style prompts. Models receive the problem in text and must provide a structured answer, often with concise reasoning.\n\n## How it is scored\n\nOpenCompass configures nine task groups and gives each a task-specific evaluator. The README labels the reported aggregate as weighted accuracy. Because the groups use different answer formats and evaluators, an aggregate score should not be treated as a single uniform item accuracy without checking the task breakdown.\n\n## Dataset and licence\n\nThe official repository describes 900 questions and is licensed Apache-2.0. OpenCompass loads local directories for hard GCP, TSP, and MSP tasks; decision variants for GCP, TSP, and KSP; and path variants for BSP, EDP, and SPP. A split protocol was not established.\n\n## Who publishes it\n\nNPHardEval is published by CASM Lab and described in the paper by Fan and colleagues, arXiv:2312.14890. OpenCompass maintains an integration configuration. No current public leaderboard was established.\n\n## Lineage\n\nThis is a family page because the official integration exposes nine named task variants: `hard_GCP`, `hard_TSP`, `hard_MSP`, `cmp_GCP_D`, `cmp_TSP_D`, `cmp_KSP`, `p_BSP`, `p_EDP`, and `p_SPP`. No predecessor or successor was established.\n\n## Saturation and contamination\n\nThe evaluation remains open in the available evidence. The public README includes a single historical model table, but no current ceiling analysis. Public algorithmic instances create medium contamination risk; no controlled refresh or exposure study was found.\n\n## How to run it\n\nUse OpenCompass\u2019s `NPHardEval` configuration with zero-shot retrieval and generation. It uses task-specific evaluators and a structured prompt format. Record the exact variant, prompt, and evaluator because variants are not interchangeable.\n\n## Reading the numbers\n\nA high score indicates that a model can solve or correctly classify the represented algorithmic instances. It does not prove a general complexity-theory understanding or reliable performance on larger unseen instances. Inspect per-variant results and answer validity, especially when comparing reasoning traces. Tool use and external computation can change the task substantially.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "nphardeval",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "reasoning",
  "contamination": {
   "note": "Public problem instances and solutions may enter training data; no exposure study was established.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "license": "Apache-2.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 900,
   "size_note": "The official OpenCompass README describes 900 algorithmic questions; the repository configuration exposes nine task groups.",
   "splits": "",
   "url": "https://github.com/casmlab/NPHardEval"
  },
  "freshness": {
   "luna-batch-017": null,
   "luna-batch-017 (Codex coordinated)": null,
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna",
   "reviewed": "2026-09-08",
   "reviewed_by": "GPT-5.6 Luna independent review"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "NPHardEval",
   "other": ""
  },
  "id": "nphardeval",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "hard_GCP",
    "hard_TSP",
    "hard_MSP",
    "cmp_GCP_D",
    "cmp_TSP_D",
    "cmp_KSP",
    "p_BSP",
    "p_EDP",
    "p_SPP"
   ]
  },
  "measures": "NPHardEval evaluates reasoning about graph, routing, matching, and path problems. Its official description says the questions span complexity classes below and through NP-hard problems.",
  "metric": {
   "baseline_note": "",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "weighted accuracy",
   "random_baseline": null,
   "unit": "percent"
  },
  "name": "NPHardEval",
  "page_kind": "family",
  "paper": {
   "arxiv": "2312.14890",
   "title": "NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes",
   "url": "https://arxiv.org/abs/2312.14890",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Lizhou Fan",
    "Wenyue Hua",
    "Lingyao Li",
    "Haoyang Ling",
    "Yongfeng Zhang",
    "Libby Hemphill"
   ],
   "org": "CASM Lab",
   "url": "https://github.com/casmlab/NPHardEval"
  },
  "released": "2023-12",
  "repo_url": "https://github.com/casmlab/NPHardEval",
  "saturation": {
   "as_of": "",
   "but no current top score was established.": null,
   "note": "The official page reports low scores for one model",
   "status": "open",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass NPHardEval README",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/NPHardEval/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass NPHardEval configuration",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/NPHardEval/NPHardEval_gen_22aac5.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "NPHardEval paper",
    "url": "https://arxiv.org/abs/2312.14890"
   }
  ],
  "status": "active",
  "subcategory": "algorithmic reasoning",
  "summary": "NPHardEval tests large language models on 900 algorithmic questions spanning complexity classes through NP-hard problems.",
  "tags": [
   "reasoning",
   "algorithms",
   "np-hard"
  ],
  "task_format": "Zero-shot natural-language problem with structured final answers and brief reasoning."
 }
}