OpenCompass wrap of IWSLT 2017 TED English-to-German: generate German from English and score with sacreBLEU, using one BM25 in-context example.
unassessed
| Category | translation |
|---|---|
| Subcategory | OpenCompass TED English-to-German text translation (sacreBLEU, BM25 1-shot) |
| Page status | active |
| Metric | BLEU (Hugging Face evaluate sacrebleu via BleuEvaluator) |
| Direction | higher_is_better |
| Dataset size | 8079 |
| Dataset licence | CC-BY-NC-ND-4.0 |
| Publisher | IWSLT 2017 organizers (FBK, KIT, NAIST, Microsoft); OpenCompass wrap by OpenCompass Authors |
OpenCompass iwslt2017 is not the full 2017 IWSLT campaign. The configs load Hugging Face `iwslt2017` with name `iwslt2017-en-de` and ask the model to translate English TED-style sentences into German. The 2017 workshop also ran multilingual many-to-many TED translation (including zero-shot pairs), a dialogue task, and a lecture ASR+MT task. Those other tracks are not what this id runs. Text in, German text out. Speech is not used. [promptbench](promptbench.md) later wraps the same en-de pair under adversarial instructions; that is a different OpenCompass directory.
Generation with BM25Retriever ice_num=1. Default config iwslt2017_gen.py re-exports iwslt2017_gen_d0ebd1: HUMAN prompt "Please translate the following English statements to German:\n{en}" and BOT target `{de}`. Two other prompt files exist (bare `{en} = {de}`, and a SYSTEM-role variant). reader_cfg uses input_columns en, output_column de, train_split validation. DatasetReader still scores the default test split.
No model card in ModelSpec reports this benchmark yet.