10 native-Italian NLP tasks, mostly drawn from the long-running Evalita campaign, each scored across six or two prompt variants to measure a model's sensitivity to prompt wording.
unassessed
| Category | composite |
|---|---|
| Subcategory | 10-task, native-Italian evaluation suite (6 multiple-choice, 4 generative), each task scored across multiple prompts |
| Page status | active |
| Metric | task-dependent (accuracy, F1/F1-macro, ROUGE), each reported per prompt and averaged across a task's prompt variants |
| Direction | higher_is_better |
| Unit | % |
| Dataset licence | Mixed by task. Checking the `evalitahf` Hugging Face organisation's own dataset cards directly found CC BY-NC-SA 4.0 stated for nine of ten component tasks checked (admission_test, faq, sentiment_analysis, hatespeech_detection, textual_entailment, entity_recognition, lexical_substitution, relation_extraction, summarization-fp); the word_in_context card sets no licence tag. No single licence file covers the composite benchmark as a whole. |
| Publisher | Fondazione Bruno Kessler (FBK), Trento, with iGenius and the Evalita organizing initiative (University of Turin coordinates Evalita's open-licence release via the Evalita4ELG project) |
Evalita-LLM tests a model on ten distinct Italian-language NLP tasks, chosen to be natively Italian rather than translated from English (avoiding both translation artefacts and cultural mismatch): word sense disambiguation (Word in Context), semantic inference (Textual Entailment), two text classification tasks (Sentiment Analysis, Hate Speech Detection), two question-answering tasks (FAQ retrieval over public-administration documents, and multiple-choice Admission Tests on scientific content), and four generative tasks -- Lexical Substitution, Named Entity Recognition (across news, literary and political-writing text), Relation Extraction, and Summarization. Most tasks are reused from earlier editions of Evalita, a biennial Italian NLP evaluation campaign running since 2007; two (Admission Tests and Summarization) were built specifically for this LLM-focused release.
Six multiple-choice-style tasks (WiC, TE, SA, HS, FAQ, Admission Tests) ask the model to select an option, scored by loglikelihood/perplexity; four generative tasks (Lexical Substitution, Named Entity Recognition, Relation Extraction, Summarization) ask the model to produce free text, scored by generate-until with task-specific parsing. The benchmark's distinguishing feature is that every task is evaluated under several different prompt phrasings of the same underlying instruction -- six templates for multiple-choice tasks, described by the paper as four templates for generative tasks -- specifically to separate genuine task competence from sensitivity to how a prompt happens to be worded.
No model card in ModelSpec reports this benchmark yet.