{
 "body": "\n## What it measures\n\n`melt_synthetic_reasoning` is HELM's Vietnamese LIME-style probe. The generator builds a pattern from rule symbols X, Y, Z and math symbols, then fills those symbols with short Vietnamese animal or fruit phrases. In pattern_match, the model sees four candidate rules and one substituted result and must emit the true rule. In variable_substitution, it applies a \"b\u1edfi \u2026\" dictionary to a rule. In induction, it sees two substituted results and must recover the rule. The language of the fillers is Vietnamese. The skill is symbol manipulation, not Vietnamese world knowledge.\n\nschema_melt.yaml groups this under \"MELT Scenarios\" and still describes that group as medical-domain work. The opened task code is Vietnamese synthetic reasoning. This page follows the task code.\n\n## How it is scored\n\nScenario metadata and schema_melt.yaml set `quasi_exact_match` on the test split. The run spec attaches HELM exact-match metrics and Vietnamese generative-harms metrics. A score is a fraction in 0-1. Pattern-match items have four candidates, but no official random baseline is given. No human baseline is given. Easy to mix this column with [melt_srn](melt_srn.md) `f1_set_match`.\n\n## Dataset and licence\n\nThere is no download. `MELTSyntheticReasoningScenario` draws 1,000 train, 5,000 validation and 5,000 test puzzles per mode with seed 42. HELM scores the 5,000 test items. The generator lives in HELM, licensed Apache-2.0. A separate dataset licence is not stated. Hugging Face mirrors used by other MELT tasks are not used here.\n\n## Who publishes it\n\nStanford CRFM ships the generator with HELM's MELT scenarios. The file cites LIME (Wu, Rabe, Li, Ba, Grosse, Szegedy; ICML 2021, arXiv:2101.06223). HELM authors of the Vietnamese wrapper are not listed in the Python file. Truong et al. (NAACL Findings 2024) published a 10-task Vietnamese LLM evaluation suite; the opened HELM scenario does not cite that paper, and the expansion of \"MELT\" is not stated in the schema. No MELT leaderboard URL was found in the opened schema or README-adjacent files.\n\n## Lineage\n\nThis is not the English HELM task [synthetic_reasoning](synthetic_reasoning.md), which uses English animal and fruit fillers and English prompt nouns. It is not [melt_srn](melt_srn.md) (`melt_synthetic_reasoning_natural`), which writes Vietnamese if-then sentences and scores set-overlap F1. The scenario class `name` is `synthetic_reasoning`, the same string as the English class; runnable names still start with `melt_synthetic_reasoning:mode=`. Sibling MELT file-slugs include [melt_ir](melt_ir.md), [melt_knowledge](melt_knowledge.md) and [melt_translation](melt_translation.md).\n\n## Saturation and contamination\n\nSaturation is unknown. Contamination risk is low for the Vietnamese strings, which are produced at evaluation time from a tiny vocabulary. English LIME pretraining or HELM's English generator could still leak the pattern.\n\n## How to run it\n\n`run_entries_melt.conf` lists:\n\n```\nmelt_synthetic_reasoning:model=text_code,mode=pattern_match\nmelt_synthetic_reasoning:model=text_code,mode=variable_substitution\nmelt_synthetic_reasoning:model=text_code,mode=induction\n```\n\nFive in-context examples come from the generated train split. max_tokens is 50. Do not report a score as `melt_synthetic_reasoning` without the mode. The natural-language runs are a different spec.\n\n## Reading the numbers\n\nA high quasi-exact match means the model copied the LIME pattern in Vietnamese fillers, often from the five-shot prompt. It is not proof of Vietnamese reading, medical knowledge, or multi-step proof. Compare modes separately, and pair a strong score with a non-synthetic Vietnamese set. Do not average it with [melt_srn](melt_srn.md).\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "melt_synthetic_reasoning",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "melt_synthetic_reasoning_pattern_match",
   "melt_synthetic_reasoning_variable_substitution",
   "melt_synthetic_reasoning_induction"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "Items are generated inside HELM from a fixed seed and a small Vietnamese vocabulary. They are not a crawled labelled test set. A model could still have seen English LIME or HELM's English synthetic_reasoning generator.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "vi"
   ],
   "license": "Apache-2.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 5000,
   "size_note": "MELTSyntheticReasoningScenario generates 1,000 train, 5,000 validation and 5,000 test instances per mode with numpy RandomState(42). size is the 5,000-item test split HELM scores. Three modes exist, so a full three-mode run scores 15,000 test items. There is no downloaded corpus. Tokens come from hardcoded ANIMALS and FRUITS lists in melt_synthetic_reasoning_scenario.py.\n",
   "splits": "generated train 1,000 / validation 5,000 / test 5,000 per mode (seed 42)",
   "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/melt_synthetic_reasoning_scenario.py"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-058 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-058"
  },
  "harness": {
   "bigbench": "",
   "helm": "melt_synthetic_reasoning:mode=pattern_match|variable_substitution|induction",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Source file helm.benchmark.scenarios.melt_synthetic_reasoning_scenario. Class name field is synthetic_reasoning, the same string as the English scenario. Distinct from melt_synthetic_reasoning_natural (melt_srn).\n"
  },
  "id": "melt_synthetic_reasoning",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "melt_srn"
   ]
  },
  "measures": "melt_synthetic_reasoning is HELM's Vietnamese abstract-symbol reasoning scenario, inspired by LIME (Wu et al., 2021). Each item is built from shuffled rule symbols X/Y/Z and math symbols +,-,*,=, a substitution into Vietnamese animal and fruit phrases, and the string after substitution. Three modes exist: pattern_match (pick the matching rule from four candidates), variable_substitution (apply a dictionary to a rule), and induction (recover the rule from two substituted results). Vietnamese words are fillers. The symbols are synthetic.\n",
  "metric": {
   "baseline_note": "schema_melt.yaml and get_metadata set main_metric / main_name quasi_exact_match on test. get_melt_synthetic_reasoning_spec uses get_exact_match_metric_specs() plus Vietnamese generative-harms metrics. Report which column you read. Pattern-match items have four candidate rules; no official random baseline is stated. No human baseline is given.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1,
   "name": "quasi_exact_match (schema); run spec attaches exact-match metrics",
   "random_baseline": null,
   "unit": ""
  },
  "name": "HELM MELT synthetic reasoning (abstract symbols)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2101.06223",
   "title": "LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning",
   "url": "https://arxiv.org/abs/2101.06223",
   "year": 2021
  },
  "publisher": {
   "authors": [],
   "org": "Stanford CRFM (HELM MELT scenarios)",
   "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/melt_synthetic_reasoning_scenario.py"
  },
  "released": "",
  "repo_url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/melt_synthetic_reasoning_scenario.py",
  "saturation": {
   "as_of": "",
   "note": "No MELT leaderboard URL resolved from opened HELM files. Saturation on the Vietnamese generated set is not established. The English HELM synthetic_reasoning paper figures are a different generator and language.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM melt_synthetic_reasoning_scenario.py",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/melt_synthetic_reasoning_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM melt_run_specs.py (melt_synthetic_reasoning)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/run_specs/melt_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM schema_melt.yaml (quasi_exact_match; medical-domain group blurb)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/static/schema_melt.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM run_entries_melt.conf",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/presentation/run_entries_melt.conf"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM Apache License 2.0",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "LIME (Wu et al., 2021), arXiv:2101.06223",
    "url": "https://arxiv.org/abs/2101.06223"
   },
   {
    "accessed": "2026-09-08",
    "title": "English HELM synthetic_reasoning_scenario.py (sibling generator)",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/synthetic_reasoning_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "Truong et al., NAACL Findings 2024 Vietnamese LLM evaluation (not cited by the scenario file)",
    "url": "https://aclanthology.org/2024.findings-naacl.182/"
   }
  ],
  "status": "unknown",
  "subcategory": "Vietnamese LIME-style pattern match, substitution and induction",
  "summary": "HELM MELT's Vietnamese LIME-style tasks: match a pattern, substitute variables, or induce a rule over abstract symbols filled with Vietnamese words.",
  "tags": [
   "synthetic",
   "reasoning",
   "vietnamese",
   "helm",
   "melt"
  ],
  "task_format": "Open generation. Run spec melt_synthetic_reasoning:mode={pattern_match, variable_substitution, induction}. Instruction \"H\u00e3y gi\u1ea3i b\u00e0i to\u00e1n sau.\", input noun B\u00e0i to\u00e1n, output noun L\u1eddi gi\u1ea3i, five in-context examples, stop at newline, max_tokens 50. HELM main_split is test.\n"
 }
}