HELM scenario: given natural-language conditional rules and facts, deduce the correct consequent, at easy/medium/hard abstraction levels.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | procedurally-generated rule-based deduction, stated in natural-language sentences |
| Page status | active |
| Metric | f1_set_match (also reports iou_set_match and exact_set_match) |
| Direction | higher_is_better |
| Publisher | Stanford CRFM (HELM) |
The model is given a small set of natural-language conditional rules (if a thing has certain attributes, it has certain other attributes) and a set of facts about specific subjects, then must deduce which consequent facts logically follow, generating its answer as text. Unlike the companion `synthetic_reasoning` scenario, which states rules and facts in an abstract symbolic notation, this variant wraps the same underlying deduction problem in simple natural-language sentences, testing whether a model can pattern-match and chain rules when they are phrased linguistically rather than symbolically. Three difficulty levels vary how abstract the subjects and attributes are (concrete named things vs. abstracted placeholders), probing whether performance depends on surface familiarity rather than the underlying logical structure.
Free-form generation: model must output the set of facts that follow from the given rules; graded as a set-matching problem, not multiple choice.
No model card in ModelSpec reports this benchmark yet.