Synthetic reasoning (HELM, abstract symbols)

HELM LIME-style synthetic tasks: match a pattern, substitute variables, or induce a rule over abstract symbols.

Also known as: HELM synthetic reasoning, synthetic_reasoning (symbolic), Synthetic reasoning (abstract symbols)

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	reasoning
Subcategory	LIME-style pattern match, variable substitution and rule induction over abstract symbols
Page status	unknown
Metric	quasi_exact_match (schema); run spec attaches exact-match metrics
Direction	higher_is_better
Dataset size	5000
Dataset licence	Apache-2.0
Publisher	Stanford CRFM (HELM)

What it measures

synthetic_reasoning is HELM's abstract-symbol reasoning scenario, inspired by LIME (Wu et al., 2021). Each item is built from a shuffled pattern of rule symbols X/Y/Z and math symbols +,-,*,=, a substitution dictionary into animal and fruit words, and the string after substitution. Three modes exist in code: pattern_match (pick the matching rule from four candidates), variable_substitution (apply a dictionary to a rule), and induction (recover the rule from two substituted results). English words appear only as substitution fillers. The symbols are synthetic.

Task format

Open generation. Run spec synthetic_reasoning:mode={variable_substitution, pattern_match, induction}. Instruction "Please solve the following problem.", output noun Target, 5 in-context examples, stop at newline, max_tokens 50. HELM main_split is test.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub