A 541-prompt professional Catalan translation of IFEval, checked by a separately reimplemented Catalan instruction-verification codebase rather than IFEval's English checkers.
unassessed
| Category | instruction-following |
|---|---|
| Subcategory | Catalan translation of IFEval's verifiable-instruction suite |
| Page status | active |
| Metric | instruction-following accuracy (strict and loose, at the prompt level and the instruction level) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 541 |
| Dataset licence | CC BY 4.0 |
| Publisher | Barcelona Supercomputing Center (BSC-CNS), Language Technologies Unit -- Projecte AINA |
ifeval_ca is Projecte AINA and the Barcelona Supercomputing Center's professional Catalan translation of Google's IFEval: 541 prompts, matching the 541-row English source ("train" split of google/IFEval) one for one. Rather than translating only the prompt text and reusing IFEval's English checking code, lm-evaluation-harness's multilingual implementation ships a dedicated Catalan instruction registry that reimplements every checker, using `langdetect` language detection and Unicode-aware case folding so Catalan text is judged correctly rather than by rules tuned for English. The registry defines 30 Catalan instruction types in code -- the original 25 English types (the two Latin-alphabet capitalisation checks become Catalan-alphabet versions) plus 5 new ones (two extra punctuation checks, three "special character" checks) -- but the 541 released prompts only ever invoke the original 25, confirmed by reading every prompt's instruction_id_list directly; the 5 unused checks, on inspection, test for the Spanish letter "n with a tilde" and Spanish-only acute accents rather than Catalan's grave accents (a/e/o with a grave accent) or c-cedilla, reading as an unadapted carry-over from the Spanish sibling file.
A model is given a Catalan prompt containing one or more verifiable instructions and generates a free-form Catalan response in a single turn, zero-shot. lm-evaluation-harness's `ifeval_ca` task generates greedily (do_sample: false) with a 1,280-token cap, then a Catalan-specific verification function checks the response against each instruction mechanically, exactly mirroring the English `ifeval` task's process_results structure.
No model card in ModelSpec reports this benchmark yet.