{
 "body": "## What it measures\n\nThe `wmt2016` directory in EleutherAI's lm-evaluation-harness contains a single runnable task, `wmt-ro-en-t5-prompt`, which evaluates English-to-Romanian machine translation on data from the WMT16 shared task. The model is given a source English sentence in a fixed prompt (\"translate English to Romanian: {sentence}\") and must generate the Romanian translation, following the prompt style originally used to evaluate T5. Despite the directory name, this task covers only the Romanian-English pair, not the full multi-language WMT16 shared task.\n\n## How it is scored\n\nThe harness computes two metrics from the generated text against the reference translation: BLEU (higher is better) and word error rate, WER (lower is better). BLEU is the metric conventionally reported for WMT tasks and is treated as primary on this page; scores that report only WER, or only BLEU, are not directly comparable without checking which metric was used.\n\n## Dataset and licence\n\nData comes from the WMT16 Romanian-English parallel corpus, hosted on Hugging Face as the `ro-en` configuration of the `wmt/wmt16` dataset, itself compiled from the WMT16 shared task (Europarl, news and other parallel sources). The harness yaml uses the `train` and `validation` splits; a held-out test split's use was not confirmed from the yaml read for this page. Hugging Face's dataset-card metadata lists the licence as \"unknown,\" and no separate licence statement was found in the task README.\n\n## Who publishes it\n\nThe Conference (formerly Workshop) on Machine Translation publishes the underlying WMT16 shared task and its findings paper. EleutherAI's lm-evaluation-harness maintains this specific `wmt-ro-en-t5-prompt` integration; no dedicated public leaderboard for this exact harness task was found.\n\n## Lineage\n\nWMT2016 sits in the annual WMT shared-task series; [WMT 14](wmt_14.md) is an earlier year covered separately in this repository via a different harness (HELM). Later WMT years exist but do not yet have pages here. The harness directory's single T5-style prompt variant is one of several ways WMT16 data has been packaged for LLM evaluation elsewhere (e.g. HELM's own WMT scenarios), which are not the same runnable task as this one.\n\n## Saturation and contamination\n\nNo saturation status was established; WMT BLEU scores for modern instruction-tuned LLMs were not found in a source opened for this page. The underlying parallel corpus has been public since 2016 and predates most current model training cutoffs, but no contamination study specific to this harness task was found.\n\n## How to run it\n\nRun the `wmt-ro-en-t5-prompt` task (not `wmt2016`, which is only the containing directory name) in lm-evaluation-harness. The task uses `generate_until` with the T5-style translation prompt described above and reports both BLEU and WER; record which metric and which harness commit were used, since prompt wording changes can shift BLEU substantially for translation tasks.\n\n## Reading the numbers\n\nA strong BLEU score on this task indicates fluent, reference-close English-to-Romanian translation under one specific prompt template, not general multilingual translation ability. Because the harness directory has historically held only this one language pair and prompt style, do not assume a \"WMT2016\" score covers other language pairs from the original shared task. Compare scores only when the metric (BLEU vs. WER) and harness revision match, and treat any score without a stated prompt template as unverified.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "wmt2016",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "translation",
  "contamination": {
   "note": "The WMT16 parallel corpus is a long-public dataset assembled from Europarl, news and other web sources predating most model training cutoffs; a formal contamination study was not found.",
   "risk": "unknown"
  },
  "dataset": {
   "languages": [
    "en",
    "ro"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": null,
   "size": null,
   "size_note": "Train and validation splits from the WMT16 Romanian-English parallel corpus; exact item counts were not read from the source.",
   "splits": "train, validation (per the harness yaml)",
   "url": "https://huggingface.co/datasets/wmt/wmt16"
  },
  "freshness": {
   "researched": "2026-09-09",
   "researched_by": "GPT-5.6 Luna, luna-stream-b-001 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, luna-stream-b-001"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "wmt-ro-en-t5-prompt",
   "opencompass": "",
   "other": ""
  },
  "id": "wmt2016",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "The lm-evaluation-harness wmt2016 task directory scores English-to-Romanian generation from the WMT16 shared-task data, prompted in the style used for T5.",
  "metric": {
   "baseline_note": "Word error rate (WER, lower is better) is also computed by the same task; BLEU is treated here as primary since it is the standard MT metric.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "BLEU",
   "random_baseline": null,
   "unit": "points"
  },
  "name": "WMT 2016 (Romanian-English, T5 prompt)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "Findings of the 2016 Conference on Machine Translation (WMT16)",
   "url": "https://aclanthology.org/W16-2301/",
   "year": 2016
  },
  "publisher": {
   "authors": [],
   "org": "",
   "url": "https://www.statmt.org/wmt16/"
  },
  "released": "2016",
  "repo_url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/wmt2016",
  "saturation": {
   "as_of": "",
   "note": "",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "wmt2016 task directory listing (lm-evaluation-harness)",
    "url": "https://api.github.com/repos/EleutherAI/lm-evaluation-harness/contents/lm_eval/tasks/wmt2016"
   },
   {
    "accessed": "2026-09-08",
    "title": "ro_en-t5_prompt.yaml task config",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/wmt2016/ro_en-t5_prompt.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "wmt2016 task README (lm-evaluation-harness)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/wmt2016/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "Findings of the 2016 Conference on Machine Translation (WMT16)",
    "url": "https://aclanthology.org/W16-2301/"
   }
  ],
  "status": "active",
  "subcategory": "machine translation",
  "summary": "The lm-evaluation-harness wmt2016 task scores Romanian-to-English translation from the WMT16 test set using a T5-style prompt.",
  "tags": [
   "benchmark",
   "machine-translation"
  ],
  "task_format": "Text generation; the model is prompted 'translate English to Romanian: {source}' and must generate the Romanian translation."
 }
}