HELM MELT synthetic reasoning (abstract symbols)

HELM MELT's Vietnamese LIME-style tasks: match a pattern, substitute variables, or induce a rule over abstract symbols filled with Vietnamese words.

Also known as: melt_synthetic_reasoning_pattern_match, melt_synthetic_reasoning_variable_substitution, melt_synthetic_reasoning_induction

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	reasoning
Subcategory	Vietnamese LIME-style pattern match, substitution and induction
Page status	unknown
Metric	quasi_exact_match (schema); run spec attaches exact-match metrics
Direction	higher_is_better
Dataset size	5000
Dataset licence	Apache-2.0
Publisher	Stanford CRFM (HELM MELT scenarios)

What it measures

melt_synthetic_reasoning is HELM's Vietnamese abstract-symbol reasoning scenario, inspired by LIME (Wu et al., 2021). Each item is built from shuffled rule symbols X/Y/Z and math symbols +,-,*,=, a substitution into Vietnamese animal and fruit phrases, and the string after substitution. Three modes exist: pattern_match (pick the matching rule from four candidates), variable_substitution (apply a dictionary to a rule), and induction (recover the rule from two substituted results). Vietnamese words are fillers. The symbols are synthetic.

Task format

Open generation. Run spec melt_synthetic_reasoning:mode={pattern_match, variable_substitution, induction}. Instruction "Hãy giải bài toán sau.", input noun Bài toán, output noun Lời giải, five in-context examples, stop at newline, max_tokens 50. HELM main_split is test.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub