IberoBench's European Portuguese suite: ASSIN entailment and paraphrase, Belebele reading, and FLORES translation, plus two extra ASSIN2 tasks in the current harness group.
unassessed
| Category | composite |
|---|---|
| Subcategory | European Portuguese IberoBench slice: NLI, paraphrase, reading comprehension and translation |
| Page status | active |
| Metric | task-dependent: accuracy (ASSIN, Belebele); BLEU/ROUGE-family scores for FLORES; macro-F1 (assin2_rte); Pearson r (assin2_sts) |
| Direction | higher_is_better |
| Dataset licence | Varies by component. facebook/belebele is CC-BY-SA-4.0. facebook/flores is CC-BY-SA-4.0 (gated). nilc-nlp/assin and nilc-nlp/assin2 list licence as unknown on their Hugging Face cards. No suite-wide licence is established. |
| Publisher | IberoBench consortium: Barcelona Supercomputing Center (BSC-CNS), Universitat Pompeu Fabra, CiTIUS-USC, and HiTZ Center - IXA (UPV/EHU) |
PortugueseBench is the European Portuguese slice of IberoBench, the same five-language project as this repository's CatalanBench, BasqueBench and GalicianBench pages. IberoBench Table 1 lists four Portuguese tasks: assin_entailment (NLI), assin_paraphrase, belebele_por_Latn (reading comprehension) and flores_pt (translation). The paper states that several task types are not yet available for European Portuguese. Translations into Portuguese were aimed at the European variety. The current lm-evaluation-harness group also runs assin2_rte and assin2_sts, which Table 1 does not list.
Mixed: two-way multiple choice for ASSIN entailment and paraphrase; multiple-choice reading for Belebele Portuguese; sentence-level machine translation for sixteen FLORES directions to or from Portuguese; multiple-choice RTE and free-form 1-5 similarity for ASSIN2 in the harness-only extras. IberoBench evaluates 0-shot and 5-shot.
No model card in ModelSpec reports this benchmark yet.