522 human-written Arabic three-choice trustworthiness questions across eight categories; HELM scores exact match on generated option letters.
unassessed
| Category | safety |
|---|---|
| Subcategory | Arabic three-choice trustworthiness QA (8 categories) |
| Page status | active |
| Metric | accuracy (paper); exact_match (HELM) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 522 |
| Dataset licence | MIT |
| Publisher | ASAS AI, with King Abdulaziz University, University College London, University of Illinois Urbana-Champaign, and Alexandria University |
AraTrust asks a model to pick the trustworthy answer to a short Arabic multiple-choice question. Authors wrote most items by hand. Eight categories cover commonsense truthfulness, ethics, physical health, mental health, unfairness, illegal activity, privacy and offensive language. It is a knowledge-and-values quiz in Arabic, not a jailbreak chat and not the later MBZUAI Arabic safeguard set (arXiv:2410.17040) that HELM's schema text incorrectly cites.
Three-option MCQ. Options in the file are prefixed with أ) ب) ج). HELM uses joint multiple-choice generation with Arabic instructions and exact_match on the letter. The paper also reports zero-shot, one-shot, few-shot and zero-shot chain-of-thought settings. One item has only two options; HELM drops empty choice fields.
No model card in ModelSpec reports this benchmark yet.