{
 "body": "\n## What it measures\n\nmelt_translation is not one dataset. It is HELM's parent class for Vietnamese\u2013English sentence\ntranslation. The model receives a source sentence and must write the translation in the other\nlanguage. Only English and Vietnamese are allowed, and one side must be English. Two run specs\nimplement that class: `melt_translation_opus100` loads the English\u2013Vietnamese slice of OPUS-100,\nand `melt_translation_phomt` loads PhoMT. OPUS-100 is a 100-language English-centric sample from\nOPUS (Zhang et al., 2020). PhoMT is VinAI's 3.02-million-pair Vietnamese\u2013English corpus with\nmanually checked validation and test sets (Doan et al., EMNLP 2021).\n\nThis is sentence MT, not document translation and not FLORES. HELM's schema labels the suite\n\"MELT Scenarios\" but describes that group as \"medical domain\", which does not match the\nVietnamese tasks listed under it. This page follows the task code. MELT here is HELM's\nVietnamese suite, not MedHELM.\n\n## How it is scored\n\nThe adapter instruction is \"Translate the following sentences from {English|Vietnamese} to\n{Vietnamese|English}.\" Default few-shot is one training sentence (`max_train_instances=1`). The\nscenario shuffles train with seed 42 and keeps at most 20,000 train rows before that one-shot\nsample. Headline metric in both ScenarioMetadata and schema_melt.yaml is `quasi_exact_match` on\ntest: a lightly normalized exact string match against the reference. The run spec also records\nexact_match, token F1, ROUGE-L, BLEU-1 and BLEU-4. Quasi-exact match is a much harsher reading\nthan the SacreBLEU of the OPUS-100 paper or the BLEU/TER of the PhoMT paper. A published BLEU\non PhoMT is not a HELM melt_translation number. No random-guess baseline applies.\n\n## Dataset and licence\n\nOPUS-100 English\u2013Vietnamese from `vietgpt/opus100_envi` (revision\n`45df06fb0b31edc882d7c8d34389261f995e5208`) has 1,000,000 train, 2,000 validation and 2,000\ntest pairs, matching Helsinki-NLP/opus-100 `en-vi` and the OPUS-100 sampling recipe. The\nvietgpt README's \"192,744 (test)\" figure is `num_bytes`, not an example count. The\nvietgpt card has no licence field. Helsinki-NLP/opus-100 tags the licence as unknown.\n\nPhoMT from `ura-hcmut/PhoMT` (revision `74386685db01dc038860ff0a90d9f5fbde284bf7`) has\n2,977,999 train, 18,720 validation and 19,151 test pairs. The EMNLP 2021 paper reports the\nsame train and test counts and 18,719 validation pairs after dropping 297 inspected-bad\nval/test rows. VinAI allows research or educational use, forbids redistribution, and requires\ncitation; the ura-hcmut card states CC BY-NC-ND 4.0. Those statements disagree. HELM's\nApache-2.0 code licence does not relicense PhoMT. Both test sets are public. HELM scores test.\n\n## Who publishes it\n\nDuc Q. Nguyen (GitHub martinakaduc) added the wrapper in HELM PR 3551, merged 22 May 2025.\nYifan Mai merged it and later added MELT scenario metadata (20 April 2026, PR 4205).\nOPUS-100 is Zhang, Williams, Titov and Sennrich (2020). PhoMT is Doan, Nguyen, Tran, Hoang\nand Nguyen at VinAI (EMNLP 2021). Stanford CRFM maintains HELM.\n`/helm/melt/latest/` returned 404 on 2026-09-08.\n\n## Lineage\n\nThis id is the HELM parent class `melt_translation`. The runnable names are\n`melt_translation_opus100` and `melt_translation_phomt`. It is not [flores](flores.md), which is\na 200-language professionally translated eval set with a hidden test split. It is not MedHELM.\nThe underlying bitext predates HELM: OPUS-100 in 2020, PhoMT in 2021. Sibling HELM MELT pages in\nthis repository are [melt_ir](melt_ir.md), [melt_knowledge](melt_knowledge.md), and\n[melt_srn](melt_srn.md). No successor wrapper under this id was found in HELM's run-spec list.\n\n## Saturation and contamination\n\nNo current top score is recorded here. Quasi-exact match on public bitext will look low next to\nBLEU even for strong MT systems, so a missing leaderboard is not evidence that the task is open\nor saturated. Both test sets have been public for years, so contamination of the exact pairs is\na real risk for models trained after 2020\u20132021. HELM pins Hugging Face revisions, which fixes\nwhich copy it loads but does not hide the sentences.\n\n## How to run it\n\nInstall HELM (`pip install crfm-helm`) and run a language-pair entry, for example\n`helm-run --run-entries melt_translation_opus100:language_pair=en-vi --suite my-suite` or\n`melt_translation_phomt:language_pair=vi-en`. There is no `@run_spec_function(\"melt_translation\")`.\nDefault is one in-context example. HELM entered maintenance mode on 1 June 2026. Compare only\nruns that share dataset, direction, shot count and headline metric. HELM's BLEU-4 is still not\nSacreBLEU as in Zhang et al.\n\n## Reading the numbers\n\nA strong quasi-exact-match score means the model often reproduced the reference after light\nnormalization. A fluent paraphrase can be a fair translation and still fail. Name the dataset\nand direction. OPUS-100 is web-mined bitext; PhoMT's test set was manually inspected. Do not\naverage the two. BLEU-4 from the same HELM run, and FLORES, are better companions. This id\ndoes not measure document-level translation or other languages.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "melt_translation",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "melt_translation_opus100",
   "melt_translation_phomt",
   "MELT OPUS100",
   "MELT PhoMT"
  ],
  "category": "translation",
  "contamination": {
   "note": "Both test sets are public. OPUS-100 has been downloadable since the 2020 paper (GitHub EdinburghNLP/opus-100-corpus; HF Helsinki-NLP/opus-100). PhoMT has been public since EMNLP 2021. HELM's wrappers pin Hugging Face revisions 45df06fb0b31edc882d7c8d34389261f995e5208 (opus100_envi) and 74386685db01dc038860ff0a90d9f5fbde284bf7 (PhoMT). Exact sentence-pair leakage into pretraining is plausible for models trained after those releases.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "vi",
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "Two separate test sets, not one pool. OPUS-100 English\u2013Vietnamese on vietgpt/opus100_envi and Helsinki-NLP/opus-100 config en-vi: 1,000,000 train / 2,000 validation / 2,000 test (Hugging Face datasets-server). That 2,000/2,000/up-to-1M sampling is the OPUS-100 paper's per-pair recipe. The vietgpt/opus100_envi README prose lists \"192,744 (test)\", which is the test split's num_bytes, not its example count. PhoMT on ura-hcmut/PhoMT: 2,977,999 train / 18,720 validation / 19,151 test (datasets-server). The PhoMT paper reports 2,977,999 train, 18,719 validation and 19,151 test after dropping 297 low-quality val/test pairs from a 3.02M-pair corpus; the HF copy is one validation row larger than the paper. HELM loads train (shuffled, capped at 20,000 rows), validation and test, then uses max_train_instances=1 by default, so the scored split is test.\n",
   "splits": "OPUS-100 en-vi: train 1,000,000 / validation 2,000 / test 2,000. PhoMT: train 2,977,999 / validation 18,720 (paper 18,719) / test 19,151. HELM scores test; few-shot examples come from train.",
   "url": "https://huggingface.co/datasets/vietgpt/opus100_envi"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-004 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-004"
  },
  "harness": {
   "bigbench": "",
   "helm": "melt_translation_opus100",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "melt_translation_phomt; parent Scenario.name is melt_translation (not a @run_spec_function). Run as melt_translation_opus100:language_pair=en-vi or vi-en, and melt_translation_phomt:language_pair=en-vi or vi-en."
  },
  "id": "melt_translation",
  "last_updated": "2026-04",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "melt_translation is HELM's Vietnamese\u2013English sentence translation wrapper. A run gives the model a source sentence in English or Vietnamese and asks it to produce the other language. The parent Scenario class only accepts the pair (vi, en) with exactly one side English. Two concrete datasets are wired: OPUS-100 English\u2013Vietnamese (Hugging Face vietgpt/opus100_envi) and PhoMT (Hugging Face ura-hcmut/PhoMT, a copy of VinAI's PhoMT). HELM's schema groups both under \"MELT Scenarios\", a Vietnamese evaluation suite contributed to HELM; this page does not treat the schema's leftover \"medical domain\" blurb as a description of these tasks. It is text-only machine translation, not FLORES and not a general multilingual MT leaderboard.\n",
  "metric": {
   "baseline_note": "Scenario metadata and schema_melt.yaml set main_metric / main_name to quasi_exact_match on the test split (lightly normalized string match, not BLEU). The run spec also attaches get_open_ended_generation_metric_specs(), which records exact_match, quasi_exact_match, f1_score, rouge_l, bleu_1 and bleu_4. That is a different headline from the OPUS-100 paper (SacreBLEU) and the PhoMT paper (BLEU and TER). No random baseline applies. PhoMT reports human preference among MT systems on 100 sampled test sentences per translation direction; that is not a HELM human baseline for this wrapper.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100.0,
   "name": "quasi_exact_match (headline); exact_match, f1_score, rouge_l, bleu_1, bleu_4 also recorded",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "MELT translation (HELM Vietnamese OPUS-100 and PhoMT)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [
    "Duc Q. Nguyen (GitHub martinakaduc; also Martin Nguyen; HELM Vietnamese translation scenarios, PR 3551)",
    "Yifan Mai (HELM maintainer; merged PR 3551 and later MELT metadata)",
    "Biao Zhang",
    "Philip Williams",
    "Ivan Titov",
    "Rico Sennrich",
    "Long Doan",
    "Linh The Nguyen",
    "Nguyen Luong Tran",
    "Thai Hoang",
    "Dat Quoc Nguyen"
   ],
   "org": "Stanford CRFM (HELM); OPUS-100 from Zhang, Williams, Titov and Sennrich; PhoMT from VinAI (Doan, Nguyen, Tran, Hoang, Nguyen)",
   "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/melt_translation_scenario.py"
  },
  "released": "2025-05",
  "repo_url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/melt_translation_scenario.py",
  "saturation": {
   "as_of": "",
   "note": "No official MELT translation leaderboard URL resolved (crfm.stanford.edu/helm/melt/latest/ returned 404 on 2026-09-08). No model card in this repository currently cites this id. Quasi-exact match on public sentence pairs is a different scale from BLEU/chrF, so a high BLEU on PhoMT is not a HELM score here.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM melt_translation_scenario.py (parent class, OPUS100 and PhoMT subclasses, pinned HF revisions, quasi_exact_match metadata)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/melt_translation_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM melt_run_specs.py (run spec names, 1-shot MT adapter, open-ended generation metrics)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/run_specs/melt_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM schema_melt.yaml (MELT groups, main_name quasi_exact_match; group description says medical domain)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/static/schema_melt.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM PR 3551, Integrate Vietnamese translation and language modeling (merged 2025-05-22)",
    "url": "https://github.com/stanford-crfm/helm/pull/3551"
   },
   {
    "accessed": "2026-09-08",
    "title": "Duc Q. Nguyen / Martin Nguyen personal page (GitHub martinakaduc)",
    "url": "https://www.comp.nus.edu.sg/~nqduc"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM commit Add metadata for MELT scenarios (#4205), 2026-04-20",
    "url": "https://github.com/stanford-crfm/helm/commit/56a5d636b09e467f30b2239726d3feeba3ba77c8"
   },
   {
    "accessed": "2026-09-08",
    "title": "vietgpt/opus100_envi dataset card (HELM OPUS-100 source; split counts vs README byte/count mix-up)",
    "url": "https://huggingface.co/datasets/vietgpt/opus100_envi"
   },
   {
    "accessed": "2026-09-08",
    "title": "datasets-server info for vietgpt/opus100_envi (train 1,000,000 / val 2,000 / test 2,000)",
    "url": "https://datasets-server.huggingface.co/info?dataset=vietgpt/opus100_envi"
   },
   {
    "accessed": "2026-09-08",
    "title": "Helsinki-NLP/opus-100 (en-vi 1,000,000 / 2,000 / 2,000; license tagged unknown)",
    "url": "https://huggingface.co/datasets/Helsinki-NLP/opus-100"
   },
   {
    "accessed": "2026-09-08",
    "title": "Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation (OPUS-100 paper)",
    "url": "https://arxiv.org/abs/2004.11867"
   },
   {
    "accessed": "2026-09-08",
    "title": "ura-hcmut/PhoMT dataset card (HELM PhoMT source; license cc-by-nc-nd-4.0 on this copy)",
    "url": "https://huggingface.co/datasets/ura-hcmut/PhoMT"
   },
   {
    "accessed": "2026-09-08",
    "title": "datasets-server info for ura-hcmut/PhoMT (2,977,999 / 18,720 / 19,151)",
    "url": "https://datasets-server.huggingface.co/info?dataset=ura-hcmut/PhoMT"
   },
   {
    "accessed": "2026-09-08",
    "title": "PhoMT: A High-Quality and Large-Scale Benchmark Dataset for Vietnamese-English Machine Translation",
    "url": "https://arxiv.org/abs/2110.12199"
   },
   {
    "accessed": "2026-09-08",
    "title": "VinAIResearch/PhoMT README (research/educational use; no redistribution; cite EMNLP 2021)",
    "url": "https://github.com/VinAIResearch/PhoMT"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM get_machine_translation_adapter_spec (instruction and prefix format)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/adaptation/common_adapter_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM get_open_ended_generation_metric_specs (exact_match, quasi_exact_match, f1, rouge_l, bleu_1, bleu_4)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/metrics/common_metric_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM README (install/run; maintenance mode from 2026-06-01)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/README.md"
   }
  ],
  "status": "unknown",
  "subcategory": "HELM Vietnamese\u2013English sentence translation (OPUS-100 and PhoMT)",
  "summary": "HELM's Vietnamese\u2013English translation pair of OPUS-100 and PhoMT, scored mainly by quasi-exact match rather than BLEU.",
  "tags": [
   "translation",
   "vietnamese",
   "helm",
   "opus-100",
   "phomt"
  ],
  "task_format": "Generate a translation of the source sentence into the target language. The adapter instruction is \"Translate the following sentences from {source} to {target}.\" with lines labelled \"English:\" / \"Vietnamese:\". Default in-context examples: 1. Language pair is a run argument (en-vi or vi-en).\n"
 }
}