{
 "body": "\n## What it measures\n\nSLR-Bench tests inductive logic programming: given a set of labelled positive and negative examples described in natural language (the released task uses classic \"trains\" compositional puzzles), a model must write a Prolog rule that correctly separates them. The task is generated automatically by the SLR framework, which for every item produces a latent ground-truth rule, an executable validation program, and the natural-language prompt shown to the model. Difficulty is controlled precisely across 20 curriculum levels of increasing relational, arithmetic and recursive complexity, and those levels are grouped into four tiers used by the released benchmark: basic, easy, medium and hard.\n\nThis page documents the group task `slr_bench_group` as implemented in lm-evaluation-harness, which combines the four difficulty tiers plus the combined all-tiers task rather than testing a single fixed task.\n\n## How it is scored\n\nCorrectness is symbolic, not string-matched: the model's generated Prolog rule is executed by a verifier (requiring SWI-Prolog) against held-out examples, and the item scores 1 if the rule classifies correctly and 0 otherwise. lm-eval's `slr_bench_group` aggregates this per-item `verifiable_reward` as a mean across five constituent tasks: the four difficulty tiers (`slr_bench_basic`, `slr_bench_easy`, `slr_bench_medium`, `slr_bench_hard`) and the combined-config task `slr_bench_all`, confirmed from the group's own YAML definition. Because scoring executes generated code through a symbolic solver, a rule that is logically equivalent to the ground truth but written differently still passes, which is stricter about correctness and looser about surface form than a typical exact-match text benchmark.\n\n## Dataset and licence\n\nHugging Face's `AIML-TUDA/SLR-Bench` dataset API lists five configs: the combined v1-All (18,053 train / 200 validation / 1,000 test) and four tier-specific configs, v1-Basic (3,053/50/250), v1-Easy, v1-Medium and v1-Hard (5,000/50/250 each). The paper's \"over 19,000 prompts\" figure is close to the All config's 19,253-row total. The dataset carries a CC BY 4.0 licence, is in English, and test splits are public with answers.\n\n## Who publishes it\n\nThe paper, \"SLR: Automated Synthesis for Scalable Logical Reasoning,\" is by Lukas Helff, Ahmad Omar, Felix Friedrich, Antonia W\u00fcst, Hikaru Shindo, Rupert Mitchell, Tim Woydt, Patrick Schramowski, Wolfgang Stammer and Kristian Kersting, first posted to arXiv in June 2025 and accepted at ACL 2026. The group is the AIML Lab at TU Darmstadt (`ml-research` on GitHub), and the benchmark and synthesis framework are released as `ScalableLogicalReasoning` / `AIML-TUDA/SLR-Bench`.\n\n## Lineage\n\nSLR-Bench is a 2025 benchmark with no predecessor tracked in this repository. Its variants inside lm-evaluation-harness are the five tasks `slr_bench_group` itself averages: the four tier tasks (`slr_bench_basic`, `slr_bench_easy`, `slr_bench_medium`, `slr_bench_hard`) and the combined-config task `slr_bench_all`; none of those individual task pages exist yet in this repository.\n\n## Saturation and contamination\n\nThe paper reports that current LLMs can produce syntactically valid Prolog rules but frequently fail at correct logical inference, and that reasoning models which do improve accuracy do so at high inference cost, exceeding $300 to evaluate 1,000 test cases for some models. That describes an open, difficulty-separating benchmark rather than a saturated one, though a specific numeric top score was not confirmed from a source opened for this page. Contamination risk is comparatively low because tasks are synthetically generated from latent rules by an automated pipeline designed to produce fresh instances at controlled difficulty, rather than scraped from public text; the specific released v1 test splits are nonetheless public, so exact-instance leakage into later training runs is possible.\n\n## How to run it\n\nlm-evaluation-harness implements this as `lm_eval/tasks/slr_bench`, with the group tag `slr_bench_group` running and averaging all five tasks in the directory: the four difficulty tiers (`slr_bench_basic`, `slr_bench_easy`, `slr_bench_medium`, `slr_bench_hard`) and the combined-config task `slr_bench_all`. Each task can also be run individually outside the group. SWI-Prolog must be installed and reachable on the system PATH for the symbolic verifier to execute generated rules. No HELM, OpenCompass, inspect_evals or BIG-bench implementation was found.\n\n## Reading the numbers\n\nA high `verifiable_reward` mean on `slr_bench_group` means a model reliably synthesises Prolog rules that are symbolically correct across a graded range of relational and recursive complexity, not merely rules that look plausible. Because scoring executes real Prolog code, a low score can reflect either weak logical reasoning or a model unfamiliar with valid Prolog syntax; check per-tier breakdowns (basic through hard) to distinguish a model that fails immediately from one that only degrades at higher curriculum levels. The paper's finding that curriculum fine-tuning on SLR data transferred to other established reasoning benchmarks suggests this benchmark is measuring a reasoning skill with some external validity, not a narrow Prolog-syntax quirk, but that transfer claim comes from the authors' own fine-tuning experiments rather than from this page's own analysis.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "slr_bench_group",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "SLR-Bench",
   "Scalable Logical Reasoning Benchmark",
   "slr_bench_group"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "Tasks are synthetically generated by the SLR framework from latent ground-truth rules rather than sourced from existing text, and the framework is designed to synthesise fresh task instances at controlled difficulty, which limits the risk of a fixed answer key leaking into pretraining corpora compared with a scraped or human-authored test set. The specific released v1 splits are nonetheless public with answers, so a model trained after the dataset's release could still have seen exact held-out instances.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC BY 4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 19303,
   "size_note": "Hugging Face's AIML-TUDA/SLR-Bench dataset API lists five configs with train/validation/test row counts: v1-All (18,053 / 200 / 1,000), v1-Basic (3,053 / 50 / 250), v1-Easy (5,000 / 50 / 250), v1-Medium (5,000 / 50 / 250), v1-Hard (5,000 / 50 / 250). All five configs' test splits (1,000 from v1-All plus 250 each from the four tiers) are the ones lm-eval's five grouped tasks draw from; the paper's \"19k+\" figure matches the All config's total of 18,053+200+1,000 = 19,253, close to but not identical to the sum across all five configs.\n",
   "splits": "train / validation / test, per curriculum tier and for the combined v1-All config",
   "url": "https://huggingface.co/datasets/AIML-TUDA/SLR-Bench"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Claude Sonnet 5, sonnet-batch-003 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, sonnet-batch-003"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "slr_bench_group",
   "opencompass": "",
   "other": "Individual lm-eval tasks: slr_bench_basic, slr_bench_easy, slr_bench_medium, slr_bench_hard, slr_bench_all (directory lm_eval/tasks/slr_bench). Requires SWI-Prolog installed and on PATH for the symbolic verifier."
  },
  "id": "slr_bench_group",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "slr_bench_basic",
    "slr_bench_easy",
    "slr_bench_medium",
    "slr_bench_hard",
    "slr_bench_all"
   ]
  },
  "measures": "SLR-Bench tests inductive logical reasoning: given natural-language-described examples (in the released task, classifying trains by their compositions), a model must produce a Prolog rule that correctly separates positive from negative examples. Tasks are automatically synthesised by the SLR framework, which generates for each item a latent ground-truth rule, an executable validation program, and the natural-language instruction prompt, at 20 curriculum levels of increasing relational, arithmetic and recursive complexity, grouped into four tiers: basic, easy, medium and hard.\n",
  "metric": {
   "baseline_note": "lm-evaluation-harness's slr_bench_group.yaml aggregates five constituent tasks (slr_bench_all, slr_bench_basic, slr_bench_easy, slr_bench_medium, slr_bench_hard) into a single mean of the verifiable_reward metric, which is 1 for a symbolically correct rule and 0 otherwise per item. No random baseline is meaningful for free-form Prolog rule generation, and no separate human baseline was read from the paper.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1.0,
   "name": "verifiable_reward (mean)",
   "random_baseline": null,
   "unit": ""
  },
  "name": "SLR-Bench (group)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2506.15787",
   "title": "SLR: Automated Synthesis for Scalable Logical Reasoning",
   "url": "https://arxiv.org/abs/2506.15787",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Lukas Helff",
    "Ahmad Omar",
    "Felix Friedrich",
    "Antonia W\u00fcst",
    "Hikaru Shindo",
    "Rupert Mitchell",
    "Tim Woydt",
    "Patrick Schramowski",
    "Wolfgang Stammer",
    "Kristian Kersting"
   ],
   "org": "AIML Lab, TU Darmstadt (with hessian.AI and DFKI)",
   "url": "https://github.com/ml-research/ScalableLogicalReasoning"
  },
  "released": "2025-06",
  "repo_url": "https://github.com/ml-research/ScalableLogicalReasoning",
  "saturation": {
   "as_of": "",
   "note": "The paper reports that contemporary LLMs produce syntactically valid Prolog rules but often fail on correct logical inference, and that advanced reasoning models improve accuracy only at high inference cost (over $300 to run 1,000 test cases with some models). A specific top-model verifiable_reward figure was not extracted from a source opened for this page, so a numeric ceiling is not recorded here, but the qualitative finding is of an unsaturated, difficulty-separating benchmark rather than one near its ceiling.\n",
   "status": "open",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "SLR paper abstract (arXiv:2506.15787), 19k-prompt benchmark, 20 curriculum levels, symbolic verification, ILP tasks",
    "url": "https://arxiv.org/abs/2506.15787"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness slr_bench README (train classification task, five task variants, SWI-Prolog requirement, AIML-TUDA/SLR-Bench dataset)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/slr_bench/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness slr_bench directory listing (confirms slr_bench_group.yaml plus basic/easy/medium/hard/all task yamls)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/slr_bench"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval slr_bench_group.yaml (group tag slr_bench_group, task list of five: slr_bench_all, slr_bench_basic, slr_bench_easy, slr_bench_medium, slr_bench_hard; verifiable_reward mean aggregation)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/slr_bench/slr_bench_group.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face dataset API for AIML-TUDA/SLR-Bench (CC-BY-4.0 licence, five configs with train/validation/test counts)",
    "url": "https://huggingface.co/api/datasets/AIML-TUDA/SLR-Bench"
   }
  ],
  "status": "active",
  "subcategory": "inductive logic programming, curriculum-graded",
  "summary": "Over 19,000 auto-synthesised inductive-logic-programming tasks across 20 curriculum levels, symbolically verified; the lm-evaluation-harness group averages verifiable-reward across five constituent tasks.",
  "tags": [
   "logic",
   "inductive-reasoning",
   "prolog",
   "curriculum",
   "symbolic-verification"
  ],
  "task_format": "Free-form generation of a Prolog rule from a natural-language prompt plus positive/negative example descriptions. Correctness is checked by executing the model's rule against a symbolic verifier (built on SWI-Prolog) rather than by string or exact-match comparison, so a semantically correct rule in a different surface form still scores as correct.\n"
 }
}