{
 "body": "\n## What it measures\n\nFLORES-200 measures machine translation quality across a huge span of languages using one\nshared, professionally translated sentence set. A system is given the same held-out sentences\nin a chosen source language and must translate them into a chosen target language; because\nevery one of the 200 languages shares identical underlying sentence content, any two language\npairs are directly comparable in a way ad hoc bilingual test sets are not. This repository's\nfour language-pair subsets (`flores_en_de`, `flores_en_es`, `flores_en_ja`, `flores_en_zh`)\neach score one English-source direction; FLORES itself supports translation between any pair\nof its 200 languages, including thousands of pairs that involve no English at all.\n\n## How it is scored\n\nThe FLORES/NLLB team's own recommendation is chrF++ as the primary automatic metric, with\nspBLEU (BLEU computed over a shared multilingual SentencePiece tokenizer, designed to be\nfairer across scripts and morphologies than word-level BLEU) reported alongside it. The two\ncorrelate highly but are not interchangeable. This repository's own model cards report plain\nBLEU in the English-source direction, sourced from a third-party aggregator rather than\ncomputed in-house \u2014 a different metric from either of FLORES' own recommended pair, so scores\nhere should not be compared numerically to chrF++ or spBLEU figures quoted elsewhere without\nknowing which metric was used.\n\n## Dataset and licence\n\nFLORES-200 consists of 3,001 sentences, drawn from 842 distinct web articles on Wikimedia\nprojects (predominantly Wikinews), each translated by professional translators into all 200\nsupported languages; many lower-resource languages were translated via a pivot language\n(Spanish, French, Russian or Modern Standard Arabic) rather than directly from English. The\nsentences split into dev (997, public), devtest (1012, public) and test (roughly 992, held out\nand not publicly released). The dataset is licensed CC BY-SA 4.0.\n\n## Who publishes it\n\nFLORES-200 was released by Meta AI's No Language Left Behind (NLLB) project, described in \"No\nLanguage Left Behind: Scaling Human-Centered Machine Translation\" (arXiv, July 2022), credited\nto the NLLB Team, a roughly 39-author group. Meta stepped back from active maintenance\nafterward; the dataset and repository are now maintained by the community-run Open Language\nData Initiative (OLDI) under the name FLORES+, with the original facebookresearch/flores\nrepository archived in November 2024 in favour of openlanguagedata/flores and a\nHugging-Face-hosted copy.\n\n## Lineage\n\nFLORES-200 is the second generation of the FLORES line, expanding FLORES-101 (\"The Flores-101\nEvaluation Benchmark for Low-Resource and Multilingual Machine Translation,\" TACL \u2014 not yet a\npage in this repository) from 101 to 200 languages using the same underlying 3,001-sentence\ncorpus. Its current continuation is FLORES+ under OLDI. In this repository, `flores_en_de`,\n`flores_en_es`, `flores_en_ja` and `flores_en_zh` are its English-source language-pair\nsubsets; FLORES supports many more pairs, including non-English-source and non-English-target\ndirections, that do not yet have pages here.\n\n## Saturation and contamination\n\nContamination risk is high rather than merely watched: the dev and devtest splits (2,009 of\nthe 3,001 sentences) are fully public with reference translations and have been online since\n2022 or earlier, and most public benchmark numbers \u2014 including this repository's \u2014 are\ncomputed on devtest rather than the small held-out test split, since devtest is the only part\nmost model providers can score against. Saturation is harder to call at the family level\nbecause it depends on the language pair: high-resource pairs such as English-German or\nEnglish-Spanish sit far higher than genuinely low-resource pairs, which is the point of the\nbenchmark, so no single saturation status applies to FLORES as a whole.\n\n## How to run it\n\nThe reference scoring path is the FLORES/OLDI repository's own scripts, which tokenize with\nSentencePiece and score with sacrebleu's chrF++ and spBLEU implementations. FLORES-200 is not\na registered task in lm-evaluation-harness's translation task list as of access (which\ncurrently covers only WMT14/16 and IWSLT2017); some third-party harnesses advertise\nFLORES-200 support, but this repository did not confirm an exact registered task name for\neither. Numbers are hard to compare across reporters when they differ on metric (chrF++ vs\nspBLEU vs plain BLEU), tokenizer, shot count or prompting style for LLM-based translation, and\npivot-language choice for low-resource pairs.\n\n## Reading the numbers\n\nA FLORES score is only meaningful next to another score computed with the same metric,\ntokenizer and split \u2014 chrF++, spBLEU and plain BLEU are not interchangeable, and this\nrepository's cards use plain BLEU rather than either of FLORES' own recommended metrics.\nBecause devtest is fully public, near-identical scores across a single provider's closely\nrelated model versions likely reflect reuse of one estimate rather than independent fresh\nruns, so treat exact ties with some caution. Compare within one language pair and one metric,\nnot across pairs, since a lower score on a low-resource pair reflects the pair's inherent\ndifficulty as much as the model being scored.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "flores",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "FLORES",
   "Flores-200",
   "FLORES+",
   "No Language Left Behind evaluation benchmark"
  ],
  "category": "translation",
  "contamination": {
   "note": "The dev and devtest splits (2009 of 3001 sentences) are fully public with reference translations and have been online since 2022 or earlier; most public benchmark numbers, including this repository's, are computed on devtest rather than the small held-out test split, since only devtest is available to score against.",
   "risk": "high"
  },
  "dataset": {
   "languages": [],
   "license": "CC BY-SA 4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": 3001,
   "size_note": "3001 sentences translated into all 200 languages, drawn from 842 distinct web articles on Wikimedia projects (predominantly Wikinews). Split: dev 997 (public), devtest 1012 (public), test ~992 (hidden, not publicly released).",
   "splits": "dev (997), devtest (1012), test (hidden, not released)",
   "url": "https://github.com/openlanguagedata/flores"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice D",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Reference scoring lives in the FLORES/OLDI repository's own scripts (SentencePiece tokenization plus sacrebleu chrF++ and spBLEU). Confirmed absent from lm-evaluation-harness's translation task list as of access (which covers only WMT14/16 and IWSLT2017); some third-party harnesses advertise FLORES-200 support without a confirmed exact task name."
  },
  "id": "flores",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "flores_en_de",
    "flores_en_es",
    "flores_en_ja",
    "flores_en_zh"
   ]
  },
  "measures": "FLORES-200 measures machine translation quality: for a chosen source and target language, a system translates the same set of held-out sentences that professional translators produced in every one of the benchmark's 200 languages, and the output is compared to the human reference. Because every language shares the same underlying sentence set, scores are directly comparable across language pairs, including many low-resource pairs with no other public evaluation data. This repository's language-pair subsets (flores_en_de, flores_en_es, flores_en_ja, flores_en_zh) each cover one English-source direction.\n",
  "metric": {
   "baseline_note": "This repository's model cards report FLORES scores as BLEU in the English-source direction, per the internal sourcing note in scripts/enrich_multilingual_benchmarks.py (citing intlpull.com). The FLORES-200/NLLB team's own recommendation is chrF++ as the primary metric, with spBLEU as the standard secondary metric; this repository's cards use neither.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "BLEU",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "FLORES-200",
  "page_kind": "family",
  "paper": {
   "arxiv": "2207.04672",
   "title": "No Language Left Behind: Scaling Human-Centered Machine Translation",
   "url": "https://arxiv.org/abs/2207.04672",
   "year": 2022
  },
  "publisher": {
   "authors": [],
   "org": "Meta AI (FAIR), NLLB Team; now maintained by the community-run Open Language Data Initiative (OLDI)",
   "url": "https://github.com/openlanguagedata/flores"
  },
  "released": "2022-07",
  "repo_url": "https://github.com/openlanguagedata/flores",
  "saturation": {
   "as_of": "",
   "note": "Saturation depends heavily on which language pair: high-resource pairs (English-German, English-Spanish) sit far higher than genuinely low-resource pairs by design, so no single status applies to the family as a whole.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "No Language Left Behind: Scaling Human-Centered Machine Translation (arXiv)",
    "url": "https://arxiv.org/abs/2207.04672"
   },
   {
    "accessed": "2026-09-07",
    "title": "facebookresearch/flores \u2014 flores200 README",
    "url": "https://raw.githubusercontent.com/facebookresearch/flores/main/flores200/README.md"
   },
   {
    "accessed": "2026-09-07",
    "title": "facebook/flores dataset card (Hugging Face)",
    "url": "https://huggingface.co/datasets/facebook/flores"
   },
   {
    "accessed": "2026-09-07",
    "title": "openlanguagedata/flores GitHub repository (current maintainer, FLORES+)",
    "url": "https://github.com/openlanguagedata/flores"
   }
  ],
  "status": "active",
  "subcategory": "machine translation",
  "summary": "A human-translated, sentence-aligned evaluation set spanning 200 languages, used to score machine-translation quality between any language pair.",
  "tags": [
   "translation",
   "multilingual",
   "machine-translation",
   "low-resource"
  ],
  "task_format": "Translate each devtest sentence from a source language into a target language; compare machine output to the human reference with an automatic metric."
 }
}