{
 "body": "\n## What it measures\n\nACPBench-Hard tests the same seven atomic planning-reasoning skills as ACPBench -- action applicability,\nprogression, atom reachability, action reachability, plan validation, action justification and landmarks\n-- over the same style of natural-language-translated PDDL problems, but strips away the multiple-choice\nand yes/no answer options that let a model narrow down a correct answer by elimination. A model must\ninstead generate the action, fact, plan or judgment directly as open-ended text. The authors add an\neighth task, Next Action, that has no analogue in base ACPBench at all: given a planning problem, produce\nthe single correct next step toward the goal, with no candidates supplied. The motivation is that a real\nplanning system never gets to choose from a curated shortlist, so a generative test is closer to how\nthese reasoning skills actually get used.\n\n## How it is scored\n\nBecause answers are open-ended text rather than a letter or yes/no, ACPBench-Hard cannot be scored by\nstring matching against one canonical answer -- a correct plan, applicable action or landmark fact can\noften be phrased more than one valid way. The authors instead built a dedicated program-based validator\nfor each of the eight tasks. The simplest, such as Applicability and Progression, can be checked in\nconstant or linear time by comparing the generated answer against the problem's known state. The hardest\n-- Reachability, Landmarks and Next Action -- require actually invoking a planner during validation, since\ndetermining whether a given claim is correct is PSPACE-complete for these tasks in general. This makes a\nreported ACPBench-Hard score dependent on which validator implementation graded it, more so than a plain\nmultiple-choice or boolean accuracy figure would be.\n\n## Dataset and licence\n\nACPBench-Hard's generative items are hosted inside the same Hugging Face repository as base ACPBench\n(`ibm-research/acp_bench`, CDLA-Permissive-2.0), as eight additional `_gen`-suffixed configs -- one per\ntask, including the new `nexta` task -- each with 130 test rows and 40 validation rows, confirmed\ndirectly from the dataset card's split metadata. That gives 1,040 test items and 320 validation items\nacross the eight generative tasks. Items are synthesized from the same 13 PDDL domains as base ACPBench\n(Blocksworld, Logistics, Grippers, Grid, Ferry, FloorTile, Rovers, VisitAll, Depot, Goldminer, Satellite,\nSwap and Alfworld), using the same solver-driven, hand-templated translation pipeline.\n\n## Who publishes it\n\nACPBench-Hard comes from the same team as the original benchmark -- Harsha Kokel, Michael Katz, Kavitha\nSrinivas and Shirin Sohrabi at IBM Research -- posted to arXiv on 2025-03-31 and revised in February 2026.\nIt was accepted to ICLR 2026, per both the paper's own arXiv listing and IBM's GitHub news feed for the\nACPBench project, which the same team continues to maintain.\n\n## Lineage\n\nACPBench-Hard is the direct generative successor to `acp_bench`, sharing its underlying PDDL domains, its\ntranslation methodology and its Hugging Face dataset repository, while replacing multiple-choice and\nboolean questions with open-ended generation and adding the Next Action task. It has no further successor\ncatalogued in this repository.\n\n## Saturation and contamination\n\nACPBench-Hard is explicitly not saturated by the authors' own framing: they report that \"no model\noutperforms another\" consistently across its tasks, and that with a few exceptions every tested model --\nincluding o1-class reasoning models -- scores below 65%, a sharp contrast with base ACPBench's multiple-\nchoice results, where the paper's own 2-shot evaluation had o1-preview reaching 87.31% average accuracy.\nThat gap is itself evidence of how much the multiple-choice format was propping up scores on the original\nbenchmark. Contamination risk is low: like the base benchmark, items are synthesized programmatically\nfrom PDDL domains with validator-checked correct answers rather than drawn from existing text, though the\ndomains themselves are old and well known in the planning literature.\n\n## How to run it\n\nlm-evaluation-harness lists `acp_bench_hard` as a separate task in the same `acpbench` directory as\n`acp_bench`, with `gen_2shot` and `gen_2shot_with_pddl` task groups -- the latter including the raw PDDL\ndomain and problem files in context alongside the natural-language description, the former using natural\nlanguage alone. Because grading requires a dedicated, per-task validator rather than a shared scoring\nfunction, and the harder tasks require running an actual planner to check a submitted answer, reproducing\na reported score depends on using the same validator implementation the original number came from. No\nHELM, Inspect Evals, BIG-bench or OpenCompass implementation was confirmed during this research.\n\n## Reading the numbers\n\nA high ACPBench-Hard score is considerably stronger evidence of real planning-reasoning skill than a high\nACPBench score, precisely because there is no answer list to narrow down by elimination -- the model has\nto produce a specific, verifiably correct action, fact or plan fragment from scratch. Given the paper's\nown finding that most models, including strong reasoning models, score under 65% with no clear leader,\ntreat any single high score with scrutiny and check which of the eight tasks and which prompting\ncondition (with or without raw PDDL in context) it covers, since the base ACPBench multiple-choice numbers\nfor the same underlying skills run dramatically higher for the same models.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "acp_bench_hard",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "ACPBench Hard"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "Like base ACPBench, items are synthesized programmatically from formally specified PDDL domains with solver- and validator-checked correct answers, rather than drawn from existing text, so a specific graded instance is unlikely to already exist verbatim in training data. The underlying domains (Blocksworld, Logistics and similar) are old and well known in the planning literature, so a model could still have absorbed general domain strategy, but the generative answer format and program-based validators make superficial memorisation of a fixed answer key less useful here than on a multiple-choice benchmark.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CDLA-Permissive-2.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 1040,
   "size_note": "1,040 test items: 8 generative tasks (app, prog, reach, areach, val, just, land, and the new nexta) x 130 test rows each, confirmed directly from the Hugging Face dataset card's per-config split metadata for the `_gen` configs in the shared `ibm-research/acp_bench` repository; each task also has 40 validation rows (320 total), for 1,360 generative items overall. Items are synthesized from the same 13 PDDL planning domains as the base ACPBench (Blocksworld, Logistics, Grippers, Grid, Ferry, FloorTile, Rovers, VisitAll, Depot, Goldminer, Satellite, Swap and Alfworld).\n",
   "splits": "test (130/task) and validation (40/task) for each of 8 generative task configs, in the same repository as base ACPBench",
   "url": "https://huggingface.co/datasets/ibm-research/acp_bench"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 4, slice C",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "acp_bench_hard (separate task in the same acpbench directory as acp_bench; gen_2shot and gen_2shot_with_pddl groups)",
   "opencompass": "",
   "other": "Task-specific validator code and the reference dataset loader are in github.com/IBM/ACPBench, alongside base ACPBench."
  },
  "id": "acp_bench_hard",
  "last_updated": "2026-02",
  "leaderboard_url": "",
  "lineage": {
   "family": "acp_bench",
   "predecessor": "acp_bench",
   "successors": [],
   "variants": []
  },
  "measures": "ACPBench-Hard tests the same underlying skill set as ACPBench -- reasoning about action applicability, progression, reachability, action reachability, plan validation, action justification and landmarks in natural-language-translated PDDL planning problems -- but removes the multiple-choice and yes/no answer options that make the base benchmark comparatively easy to pass by elimination. A model must instead produce the answer directly as open-ended text: name the applicable action, state the resulting facts, output a plan, or identify the unjustified step, rather than pick from a short list. ACPBench-Hard adds an eighth task on top of the original seven, Next Action, which asks a model to choose the correct next step toward a goal with no candidates supplied at all. The authors motivate this generative format as a closer match to how an actual planning system operates, which never gets to choose from a curated multiple-choice menu.\n",
  "metric": {
   "baseline_note": "There is no meaningful random-guess baseline for open-ended generation the way there is for ACPBench's boolean (~50%) or four-option multiple-choice (~25%) formats. Instead of string matching, each task has its own correctness validator: simple tasks such as Applicability and Progression can be checked in constant or linear time against the problem's known state, while harder tasks -- Reachability, Landmarks and Next Action -- require actually invoking a planner, since checking them is PSPACE-complete in the general case. No controlled human baseline is published.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy (each generated answer checked by a task-specific program validator, not string matching)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "ACPBench-Hard",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2503.24378",
   "title": "ACPBench Hard: Unrestrained Reasoning about Action, Change, and Planning",
   "url": "https://arxiv.org/abs/2503.24378",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Harsha Kokel",
    "Michael Katz",
    "Kavitha Srinivas",
    "Shirin Sohrabi"
   ],
   "org": "IBM Research",
   "url": "https://ibm.github.io/ACPBench"
  },
  "released": "2025-03",
  "repo_url": "https://github.com/IBM/ACPBench",
  "saturation": {
   "as_of": "",
   "note": "No single leading score was found in the sources read for this page, and the paper's own framing argues against picking one: it reports that \"no model outperforms another\" consistently across ACPBench-Hard's tasks, and that \"with a few exceptions all tested language models score below 65%,\" a group that includes o1-class reasoning models struggling on roughly half the benchmark. That combination -- no clear leader, and a hard ceiling most models do not approach -- is the paper's own evidence that removing multiple-choice framing from ACPBench meaningfully raises its difficulty, consistent with an open rather than saturated or merely-watch benchmark.\n",
   "status": "open",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "ACPBench Hard: Unrestrained Reasoning about Action, Change, and Planning (Kokel, Katz, Srinivas, Sohrabi, 2025)",
    "url": "https://arxiv.org/abs/2503.24378"
   },
   {
    "accessed": "2026-09-08",
    "title": "ACPBench Hard, full text (ar5iv) -- task definitions, validators, results",
    "url": "https://ar5iv.labs.arxiv.org/html/2503.24378"
   },
   {
    "accessed": "2026-09-08",
    "title": "IBM/ACPBench GitHub repository -- task table (App/Prog/Reach/Val/AReach/Just/Land/NextA), news section confirming ICLR 2026 acceptance",
    "url": "https://github.com/IBM/ACPBench"
   },
   {
    "accessed": "2026-09-08",
    "title": "ibm-research/acp_bench dataset card, Hugging Face -- CDLA-Permissive-2.0 licence, per-config _gen split sizes",
    "url": "https://huggingface.co/datasets/ibm-research/acp_bench"
   },
   {
    "accessed": "2026-09-08",
    "title": "ibm-research/acp_bench dataset card API -- exact _gen config split sizes",
    "url": "https://huggingface.co/api/datasets/ibm-research/acp_bench"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness acpbench task directory (acp_bench and acp_bench_hard task names)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/acpbench"
   }
  ],
  "status": "active",
  "subcategory": "generative planning and action reasoning (open-ended variant of ACPBench)",
  "summary": "IBM's harder, generative companion to ACPBench: the same planning-reasoning skills plus a new 'next action' task, answered as open text and checked by per-task validators, not multiple choice.",
  "tags": [
   "planning",
   "reasoning",
   "action-reasoning",
   "pddl",
   "generative",
   "open-ended",
   "iclr-2026"
  ],
  "task_format": "Open-ended, free-text generation over the same natural-language-translated PDDL planning problems ACPBench uses: given a context and a question (for example, \"which action is applicable in this state?\" or \"what is a valid plan to reach the goal?\"), the model must produce the exact action, fact, action sequence or judgment in its own words, with no answer choices given. Two prompting conditions are used in lm-evaluation-harness: natural-language-only, and natural language with the underlying PDDL domain/problem file included in context.\n"
 }
}