87,869 Bangla four-choice exam questions from Bangladeshi admissions, HSC and job tests; lm-eval scores the 14,750-item test split.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | Bangla four-choice exam questions from Bangladeshi admissions, HSC and job tests |
| Page status | active |
| Metric | accuracy (acc and length-normalised acc_norm) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 87869 |
| Publisher | Hishab Singapore Pte. Ltd., with University of Central Florida and Qatar Computing Research Institute |
Bangla MMLU asks a model to pick one of four options for a Bangla multiple-choice question drawn from Bangladeshi educational and professional exams. The TituLLMs paper describes it as inspired by English MMLU, not as a translation of Hendrycks's 57-subject set. Items were curated from open educational sites and textbooks covering university admission, higher secondary, job exams, medical admission and engineering admission. The skill is Bangla-medium school and professional knowledge, including Bangladesh-specific general knowledge.
Four-option multiple-choice QA in Bangla. lm-eval task `bangla_mmlu` formats the stem plus A–D options and scores by comparing log-likelihoods of the letters. Few-shot examples come from the 175-item `dev` split (`sampler: first_n`). Headline split is `test`.
No model card in ModelSpec reports this benchmark yet.