lm-eval group of GPT-translated TruthfulQA val items in 31 languages, scored as MC1 accuracy and MC2 probability mass.
unassessed
| Category | safety |
|---|---|
| Subcategory | machine-translated multiple-choice truthfulness |
| Page status | active |
| Metric | MC1 accuracy; MC2 normalised true-answer probability (acc) |
| Direction | higher_is_better |
| Dataset size | 23586 |
| Dataset licence | CC-BY-NC-4.0 |
| Publisher | University of Oregon NLP (translations); Alexandra Institute (Hub dump); EleutherAI (lm-eval group) |
okapi_truthfulqa_multilingual is EleutherAI's group over machine-translated [TruthfulQA](truthfulqa.md). The model is asked a question written to bait a popular misconception and must prefer the true continuation over false ones. Hub dump alexandrainst/m_truthfulqa was translated with GPT-3.5-turbo by University of Oregon and first lived in nlp-uoregon/mlmm-evaluation. The Okapi paper's own eval suite is ARC, HellaSwag, and MMLU; this TruthfulQA translation uses the same pipeline but is not one of those three paper tables. English text only in the six-shot prefix; questions are in the target language.
Multiple choice, two YAML variants per language: truthfulqa_{lang}_mc1 and truthfulqa_{lang}_mc2. README group/tag truthfulqa_multilingual. MC1 target index is 0 (first listed choice). MC2 uses labelled true/false options and sums normalised probability mass on true labels. Validation split val. process_docs prepends a fixed English 6-QA prompt before the translated question.
No model card in ModelSpec reports this benchmark yet.