{
 "body": "\n## What it measures\n\nACPBench gives a model a natural-language description of a planning problem -- translated from a\nformal PDDL domain and problem definition, spanning domains such as Blocksworld, Logistics,\nGrippers, Ferry, Rovers, Satellite and Alfworld -- and asks a targeted question exercising one of\nseven atomic reasoning skills that end-to-end planning depends on: whether an action is applicable in\nthe current state, what the state looks like after an action (progression), whether a goal fact is\nreachable at all, whether an action sequence validly achieves the goal, whether an action could ever\nbecome applicable along some future path (action reachability), whether an action inside a plan is\nunnecessary and can be dropped without breaking it (justification), and which facts every valid plan\nmust pass through (landmarks). Rather than asking a model to generate a full plan -- hard to evaluate\nand hard to attribute failure within -- it isolates these component skills so a wrong answer points\nat a specific reasoning gap.\n\n## How it is scored\n\nEvery question is either boolean (yes/no) or four-option multiple-choice, scored as plain accuracy;\nthere is no free-text generation or partial credit in the base tasks. Chance level is roughly 50% for\nboolean questions and 25% for multiple-choice, though the paper reports per-task accuracy rather than\none blended figure, since difficulty varies sharply -- GPT-4o's reported accuracy ranges from 52.50%\non the hardest task (validation) up past 90% on the easiest. ACPBench-Hard, a separate, harder\ncollection, replaces the fixed answer options with open-ended generative questions and provides its\nown per-task validation algorithms rather than string matching, since a generated plan fragment can\nbe correct in more than one surface form.\n\n## Dataset and licence\n\nThe Hugging Face dataset (`ibm-research/acp_bench`, CDLA-Permissive-2.0) packages each of the seven\nACPBench tasks in both boolean and multiple-choice format, each with roughly 120-130 test rows and 40\nvalidation rows, across 13 planning domains: Blocksworld, Logistics, Grippers, Grid, Ferry, FloorTile,\nRovers, VisitAll, Depot, Goldminer, Satellite, Swap and Alfworld. The paper's Table 1 reports domain\ncomplexity (predicate and lifted-action counts) rather than a single overall item total. The top 8 of\nthese domains are also used in a fine-tuning experiment testing whether smaller models can be taught\nthese skills directly, while the remaining 5 are held out for evaluation only, to check\ngeneralisation to unseen domains. Problem instances are synthesized from the underlying PDDL domain\nfiles by a solver-driven pipeline with provably correct answers, and translated into natural language\nusing templates the authors report crafting by hand after finding LLM-generated templates unreliable.\n\n## Who publishes it\n\nACPBench was introduced by Harsha Kokel, Michael Katz, Kavitha Srinivas and Shirin Sohrabi at IBM\nResearch, posted to arXiv in October 2024 and presented at AAAI 2025. IBM continues to maintain the\nbenchmark actively: a generative follow-up, ACPBench-Hard, was accepted at ICLR 2026, and the\noriginal paper's own arXiv listing was revised as recently as February 2026. The team also presented\nthe work in a NeurIPS 2025 tutorial on planning with language models.\n\n## Lineage\n\nACPBench has no predecessor benchmark of its own, though the authors position it against a related\nprior benchmark, ActionReasoningBench, noting deliberate overlap (both cover an actions'-effects\ntask) alongside deliberate non-overlap (ACPBench adds reachability, action reachability, validation,\njustification and landmarks). Its direct successor, ACPBench-Hard (`acp_bench_hard`, not yet a\nseparate page here), extends the same seven tasks into open-ended generative questions and adds an\neighth, \"next action,\" while sharing the same Hugging Face dataset repository -- which is why that\nrepository's own description currently lists eight tasks rather than the original paper's seven.\n\n## Saturation and contamination\n\nACPBench is not saturated. In the paper's 2-shot chain-of-thought evaluation, OpenAI's o1-preview\nreached the highest reported average, 87.31% across the multiple-choice tasks, still dropping to\n63.08% on its hardest task; GPT-4o averaged 78.40%, falling to 52.50% on the hardest (validation)\ntask. The harder, generative ACPBench-Hard follow-up reports a starker picture: \"no model outperforms\nanother\" consistently, and with few exceptions every tested model -- including o1-class reasoning\nmodels -- scores below 65%. No 2025-2026 frontier-model score was found for either version.\nContamination risk is low: problems are synthesized programmatically from formal PDDL domains with\nsolver-verified answers rather than sourced from existing text, though the domains themselves\n(Blocksworld, Logistics and similar) are decades-old and widely known in the planning literature.\n\n## How to run it\n\nlm-evaluation-harness ships the benchmark under the task name `acp_bench` (in a directory named\n`acpbench`), with `boolq_cot_2shot` and `mcq_cot_2shot` task groups matching the original paper's\nboolean and multiple-choice formats, plus `gen_2shot` and `gen_2shot_with_pddl` groups for\nACPBench-Hard's generative tasks (listed as a separate `acp_bench_hard` task pulling from the same\ndataset repository). No HELM, Inspect Evals, BIG-bench or OpenCompass task was confirmed during this\nresearch. Because the generative tasks require dedicated per-task validators rather than string\nmatching -- the authors built a distinct correctness checker for each generative task type -- a\ngenerative-format score depends on which validator implementation graded it, more so than the\nboolean or multiple-choice base tasks do.\n\n## Reading the numbers\n\nA high ACPBench score is evidence a model can reliably reason about the mechanics of a planning\nproblem -- what is currently possible, what would happen next, what is reachable, what is redundant --\nin its simplest boolean or multiple-choice form; it does not show the model can generate a full\nmulti-step plan, which is a substantially harder, open-ended task that ACPBench-Hard targets instead.\nBecause per-task accuracy varies widely (the paper's own numbers span roughly 50 to 98 percentage\npoints across tasks for the same model), an aggregate ACPBench score can hide a specific weak skill --\nvalidation and action-reachability were the hardest tasks for every model in the original paper -- so\ncheck the task-level breakdown before concluding a model can plan reliably.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "acp_bench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "ACP Bench",
   "acp-bench"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "Problems are synthesized programmatically from formally specified PDDL planning domains with provably correct solutions, rather than drawn from an existing text corpus, so a specific graded instance is unlikely to pre-exist in training data verbatim; the authors note this construction method lets them scale to more problems without added human effort. The domains themselves (Blocksworld, Logistics and other classic PDDL benchmarks) are old and well known in the planning literature, so a model could still have absorbed general strategies for solving them.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CDLA-Permissive-2.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "Per-task-per-format counts on the Hugging Face dataset (ibm-research/acp_bench) run 120-130 test rows and 40 validation rows for each of the seven original ACPBench tasks in each of its boolean and multiple-choice formats (14 configs), plus generative-format configs added by the follow-on ACPBench-Hard collection that shares the same repository; the paper's Table 1 gives per-domain problem statistics across 13 domains instead of a single item total, so no single confirmed total question count is reported here. The top 8 of the 13 domains (by the paper's own grouping) are also used for a fine-tuning experiment; the remaining 5 are held out for evaluation only.\n",
   "splits": "test and validation rows per task-format config; see size_note",
   "url": "https://huggingface.co/datasets/ibm-research/acp_bench"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 2, slice C",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "acp_bench (harness directory named acpbench; boolq_cot_2shot, mcq_cot_2shot, gen_2shot and gen_2shot_with_pddl task groups)",
   "opencompass": "",
   "other": "acp_bench_hard is a separate lm-evaluation-harness task, in the same acpbench directory, covering ACPBench-Hard's generative questions."
  },
  "id": "acp_bench",
  "last_updated": "2026-02",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [
    "acp_bench_hard"
   ],
   "variants": []
  },
  "measures": "ACPBench gives a model a natural-language description of a planning problem (translated from a formal PDDL domain and problem definition, covering domains such as Blocksworld, Logistics, Grippers and Alfworld) and asks a targeted question about one of seven atomic reasoning skills: whether an action is applicable in the current state, what a state looks like after an action (progression), whether a goal atom is reachable, whether an action sequence validly achieves a goal, whether an action could ever become applicable on some future path (action reachability), whether an action in a plan is unjustified and can be dropped, or which facts every valid plan must pass through (landmarks). It isolates the specific reasoning sub-skills end-to-end plan generation depends on, rather than asking a model to produce a full plan itself.\n",
  "metric": {
   "baseline_note": "Random-guess baseline is approximately 50% for the boolean questions and 25% for the four-option multiple-choice questions; the paper reports per-task accuracy rather than a single blended figure, so no single random baseline applies to an aggregate score. No controlled human baseline is published.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 50,
   "unit": "%"
  },
  "name": "ACPBench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2410.05669",
   "title": "ACPBench: Reasoning about Action, Change, and Planning",
   "url": "https://arxiv.org/abs/2410.05669",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Harsha Kokel",
    "Michael Katz",
    "Kavitha Srinivas",
    "Shirin Sohrabi"
   ],
   "org": "IBM Research",
   "url": "https://ibm.github.io/ACPBench"
  },
  "released": "2024-10",
  "repo_url": "https://github.com/IBM/ACPBench",
  "saturation": {
   "as_of": "2024-10",
   "note": "The paper's own evaluation (2-shot chain-of-thought) has OpenAI o1-preview at 87.31% average accuracy on the multiple-choice tasks, with its weakest single task down at 63.08%; GPT-4o trails at 78.40% average, with its weakest task (validation) at 52.50%. ACPBench-Hard's generative-format follow-up (arXiv 2503.24378) reports that on its harder, open-ended tasks \"no model outperforms another\" consistently and \"with a few exceptions all tested language models score below 65%\" -- including o1-class reasoning models. No 2025-2026 frontier-model score was found during this research.\n",
   "status": "open",
   "top_score": 87.31
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "ACPBench: Reasoning about Action, Change, and Planning (Kokel et al., 2024)",
    "url": "https://arxiv.org/abs/2410.05669"
   },
   {
    "accessed": "2026-09-08",
    "title": "ACPBench paper, full text (ar5iv HTML) -- task definitions, domain table, accuracy results",
    "url": "https://ar5iv.labs.arxiv.org/html/2410.05669"
   },
   {
    "accessed": "2026-09-08",
    "title": "IBM/ACPBench GitHub repository (task table, ACPBench-Hard news, worked examples)",
    "url": "https://github.com/IBM/ACPBench"
   },
   {
    "accessed": "2026-09-08",
    "title": "ibm-research/acp_bench dataset card, Hugging Face (CDLA-Permissive-2.0, 13 domains, per-config split sizes)",
    "url": "https://huggingface.co/datasets/ibm-research/acp_bench"
   },
   {
    "accessed": "2026-09-08",
    "title": "ACPBench Hard: Unrestrained Reasoning about Action, Change, and Planning (Kokel et al., 2025) -- successor benchmark",
    "url": "https://arxiv.org/abs/2503.24378"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness acpbench task directory (acp_bench and acp_bench_hard task names)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/acpbench"
   }
  ],
  "status": "active",
  "subcategory": "planning and action reasoning",
  "summary": "IBM's boolean- and multiple-choice test of seven atomic reasoning skills needed for planning -- action applicability, reachability, justification, landmarks and more -- across 13 formal domains.",
  "tags": [
   "planning",
   "reasoning",
   "action-reasoning",
   "pddl",
   "multiple-choice",
   "boolean"
  ],
  "task_format": "Boolean (yes/no) or four-option multiple-choice questions over a natural-language-translated planning problem; the base ACPBench tasks require no free-text generation (a separate, harder generative version, ACPBench-Hard, exists -- see Lineage)."
 }
}