Over 19,000 auto-synthesised inductive-logic-programming tasks across 20 curriculum levels, symbolically verified; the lm-evaluation-harness group averages verifiable-reward across five constituent tasks.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | inductive logic programming, curriculum-graded |
| Page status | active |
| Metric | verifiable_reward (mean) |
| Direction | higher_is_better |
| Dataset size | 19303 |
| Dataset licence | CC BY 4.0 |
| Publisher | AIML Lab, TU Darmstadt (with hessian.AI and DFKI) |
SLR-Bench tests inductive logical reasoning: given natural-language-described examples (in the released task, classifying trains by their compositions), a model must produce a Prolog rule that correctly separates positive from negative examples. Tasks are automatically synthesised by the SLR framework, which generates for each item a latent ground-truth rule, an executable validation program, and the natural-language instruction prompt, at 20 curriculum levels of increasing relational, arithmetic and recursive complexity, grouped into four tiers: basic, easy, medium and hard.
Free-form generation of a Prolog rule from a natural-language prompt plus positive/negative example descriptions. Correctness is checked by executing the model's rule against a symbolic verifier (built on SWI-Prolog) rather than by string or exact-match comparison, so a semantically correct rule in a different surface form still scores as correct.
No model card in ModelSpec reports this benchmark yet.