{
 "body": "\n## What it measures\n\nThe translation group evaluates whether a model can generate text in a target language from a source sentence.\n\n## How it is scored\n\nMetrics are member-specific overall, but for the WMT-family groups (`wmt14`, `wmt16`) the shared `wmt_common_yaml` config confirms that each task reports BLEU, TER, and chrF together, with `generate_until` output and greedy decoding (temperature 0, no sampling, stopped at a newline). TER is a lower-is-better error-rate metric, so a single \"higher is better\" direction does not describe all three scores reported for one task. The `gpt3_translation_tasks`, `wmt20`, and `iwslt2017` groups were not independently confirmed to use the same metric set.\n\n## Dataset and licence\n\nThe harness README lists gpt3_translation_tasks, wmt14, wmt16, wmt20, and iwslt2017 groups. The two WMT14 tasks translate between English and French (`wmt14_en-fr`, `wmt14_fr-en`), loading the `wmt/wmt14` dataset; the WMT16 tasks cover English-German and English-Romanian pairs; IWSLT2017 covers Arabic-English. Aggregate size and a common licence across the group were not established, since each member task draws on a separately licensed WMT or IWSLT release.\n\n## Who publishes it\n\nEleutherAI maintains the integration. Member datasets retain their own publishers and papers.\n\n## Lineage\n\nThis is a harness task group, not one homogeneous dataset. Its members are variants rather than interchangeable aliases.\n\n## Saturation and contamination\n\nAggregate saturation is unknown because the group mixes datasets and language pairs.\n\n## How to run it\n\nUse the exact lm-evaluation-harness member task, language pair, dataset revision, and metric -- for example `wmt14_en-fr` rather than \"translation\" or \"wmt14\". The group name alone is insufficient for comparison, since it can refer to any of several language pairs and years. For the WMT-family tasks, the reference implementation is `lm_eval/tasks/translation/wmt_common_yaml`, which fixes greedy decoding and reports BLEU, TER, and chrF together; other harnesses (HELM, OpenCompass) run their own WMT configurations with different prompts, shot counts, and metric sets, so a \"WMT14\" score from a different tool is not directly comparable.\n\n## Reading the numbers\n\nA strong score indicates translation quality for the selected language pair and metric. It does not imply equal performance across the group. Compare language pair, reference set, decoding, and metric.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "translation",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "translation",
  "contamination": {
   "note": "Member datasets are public with different histories; suite-level contamination is unknown.",
   "risk": "unknown"
  },
  "dataset": {
   "languages": [
    "multiple"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "The harness README does not give one aggregate item count or common score.",
   "splits": "varies",
   "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/translation"
  },
  "freshness": {
   "luna-new-002": null,
   "luna-new-002 (Codex coordinated)": null,
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "translation",
   "opencompass": "",
   "other": ""
  },
  "id": "translation",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "The group covers translation evaluations including WMT14, WMT16, WMT20, and IWSLT2017 task groups.",
  "metric": {
   "baseline_note": "Confirmed from the WMT group's shared wmt_common_yaml, which sets output_type: generate_until with greedy decoding (temperature 0, no sampling) and reports bleu, ter, and chrf together. TER is conventionally lower_is_better, so a single 'higher_is_better' direction does not hold across all three metrics in the group.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "BLEU, TER, and chrF (reported together for the WMT14/WMT16 groups; other groups may differ)",
   "random_baseline": null,
   "unit": ""
  },
  "name": "Translation Tasks",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [],
   "org": "EleutherAI",
   "url": "https://github.com/EleutherAI/lm-evaluation-harness"
  },
  "released": "2023",
  "repo_url": "https://github.com/EleutherAI/lm-evaluation-harness",
  "saturation": {
   "as_of": "",
   "note": "No aggregate leaderboard was established.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness translation task README",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/translation/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness wmt14 en-fr task config",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/translation/wmt14_en-fr.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness WMT common config (metrics and generation settings)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/translation/wmt_common_yaml"
   }
  ],
  "status": "active",
  "subcategory": "machine translation",
  "summary": "A family of translation tasks configured in lm-evaluation-harness.",
  "tags": [
   "translation",
   "multilingual"
  ],
  "task_format": "Generate a translation for a source sentence."
 }
}