An lm-evaluation-harness suite of about twenty French tasks mixing native resources (FQuAD, French Trivia, OrangeSum) with GPT-3.5-translated ones (HellaSwag, ARC-Challenge), from the CroissantLLM paper.
unassessed
| Category | composite |
|---|---|
| Subcategory | native and machine-translated French suite: generative QA/summarisation, multiple-choice reasoning and linguistics, translation |
| Page status | active |
| Metric | varies by sub-task: accuracy/acc_norm for multiple-choice and classification tasks, exact-match/F1/ROUGE-1 for generative QA and summarisation, BLEU/COMET for translation |
| Direction | higher_is_better |
| Unit | % |
| Dataset licence | Varies by sub-task: manu/fquad2_test is Apache-2.0 and manu/french-bench-grammar-vocab-reading is MIT (both per their Hugging Face cards); French Trivia is separately stated in the paper as MIT-licensed and MultiFQuAD's evaluation set as CC-BY-NC-SA 4.0; several `manu/`-hosted component cards (french_boolq, topic_based_nli_test, multifquad_test, french-trivia, french_bench_hellaswag, french_bench_arc_challenge) carry no machine-readable licence tag at all. This page did not check every component individually, so one suite-wide licence is not established. |
| Publisher | CroissantLLM project: core academic authors affiliated with CentraleSupelec (Universite Paris Saclay) and Instituto Superior Tecnico de Lisboa, with further contributors at Sorbonne Universite and Imperial College London; core industrial authors funded by Illuin Technology (Paris), Unbabel (Lisboa) and Equall (New York, Lisboa, Paris) -- all per the paper's own acknowledgements. |
FrenchBench bundles roughly twenty French-language tasks into one lm-evaluation-harness group, covering generative QA and summarisation (FrenchBench Gen), multiple-choice reasoning, linguistics and knowledge (FrenchBench MC), and further extras (an XNLI French config, a French BoolQ, a topic-based sentiment task) not part of either core tagged set. Critically, the tasks are not uniformly native or translated: FQuAD-derived question answering, MultiFQuAD, French Trivia, OrangeSum summarisation and the grammar/vocabulary/reading tests were authored or curated directly in French, while French HellaSwag and French ARC-Challenge are GPT-3.5 machine translations of their English originals, and XNLI(fr) and Belebele(fr) are professionally translated multilingual sets with a French configuration. The paper's own authors are explicit that the machine-translated pair is imperfect: "manual verification of the translation quality indicates the translations to be far from perfect but sufficient... to act as a correct performance proxy."
Varies by sub-task: 4- or 5-option multiple choice for HellaSwag(fr), ARC-Challenge(fr), the grammar/vocabulary/reading tests and XNLI(fr); binary classification for French BoolQ; extractive or short free-text generation, scored few-shot, for the FQuAD-derived QA tasks and French Trivia; single-sentence or first-paragraph generation for OrangeSum; and sentence-level machine translation for the WMT14 English-French pair. lm-evaluation-harness exposes tag groups `french_bench_gen`, `french_bench_mc`, `french_bench_perplexity` and `french_bench_extra` alongside the umbrella `french_bench` tag (non-perplexity tasks only).
No model card in ModelSpec reports this benchmark yet.