OALL's 2,106-item Arabic passage QA set; HELM grades free-form answers with GPT-4o, using the public train split as the test set.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | Arabic open-book RAG question answering, GPT-4o graded |
| Page status | active |
| Metric | alrage_score |
| Direction | higher_is_better |
| Unit | 0-1 |
| Dataset size | 2106 |
| Publisher | OALL (Open Arabic LLM Leaderboard); HELM scenario by Stanford CRFM |
ALRAGE is an Arabic open-book question answering task built to look like retrieval-augmented generation. HELM shows the model an Arabic question and a block of suggested passages (`candidates`), then asks it to write an answer. The gold string is a short reference answer, not a multiple-choice letter. The language is Arabic. The skill is using the supplied passages, not browsing the web, and not the closed-book exam mix in [alghafa](alghafa.md).
Open-ended generation. HELM concatenates "السؤال:" plus the question and "السياقات المقترحة:" plus `candidates`. The generation adapter uses Arabic instructions, max_tokens 100, and no stop sequences. Scoring is an LLM judge, not string match, despite leftover exact_match metadata on the scenario class.
No model card in ModelSpec reports this benchmark yet.