IBM's harder, generative companion to ACPBench: the same planning-reasoning skills plus a new 'next action' task, answered as open text and checked by per-task validators, not multiple choice.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | generative planning and action reasoning (open-ended variant of ACPBench) |
| Page status | active |
| Metric | accuracy (each generated answer checked by a task-specific program validator, not string matching) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1040 |
| Dataset licence | CDLA-Permissive-2.0 |
| Publisher | IBM Research |
ACPBench-Hard tests the same underlying skill set as ACPBench -- reasoning about action applicability, progression, reachability, action reachability, plan validation, action justification and landmarks in natural-language-translated PDDL planning problems -- but removes the multiple-choice and yes/no answer options that make the base benchmark comparatively easy to pass by elimination. A model must instead produce the answer directly as open-ended text: name the applicable action, state the resulting facts, output a plan, or identify the unjustified step, rather than pick from a short list. ACPBench-Hard adds an eighth task on top of the original seven, Next Action, which asks a model to choose the correct next step toward a goal with no candidates supplied at all. The authors motivate this generative format as a closer match to how an actual planning system operates, which never gets to choose from a curated multiple-choice menu.
Open-ended, free-text generation over the same natural-language-translated PDDL planning problems ACPBench uses: given a context and a question (for example, "which action is applicable in this state?" or "what is a valid plan to reach the goal?"), the model must produce the exact action, fact, action sequence or judgment in its own words, with no answer choices given. Two prompting conditions are used in lm-evaluation-harness: natural-language-only, and natural language with the underlying PDDL domain/problem file included in context.
No model card in ModelSpec reports this benchmark yet.