5,944-item machine translation of OpenBookQA into Bangla, built with an automated Google-Translate-plus-LLM-rewriting pipeline the authors call Expressive Semantic Translation.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | Bangla open-book elementary-science QA, machine-translated from OpenBookQA |
| Page status | active |
| Metric | accuracy (acc and length-normalised acc_norm) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 5944 |
| Dataset licence | Not stated: the Hugging Face dataset card's licence field is set to "unknown", and the TituLLMs paper does not separately name a licence for this dataset. |
| Publisher | Hishab (Hishab Singapore Pte. Ltd), with the University of Central Florida and the Qatar Computing Research Institute |
Bangla OpenBookQA asks a model to answer a four-way multiple-choice elementary-science question in Bangla, combining one core science fact with broader common knowledge -- the same two-hop structure as the English OpenBookQA. It is a translation rather than a from-scratch Bangla benchmark: the TituLLMs paper that introduces it describes it as "a Bangla translation of the OpenBookQA dataset," produced with the authors' Expressive Semantic Translation (EST) pipeline, a standard machine-translation pass refined through iterative, automatically-ranked LLM rewrites. No human translation or per-item human verification of the finished Bangla items is described. The lm-evaluation-harness's own top-level task registry misspells this benchmark's display name as "bangla_poenbookQA" (transposing "openbook" to "poenbook"); that misspelling is a typo in the harness's own documentation, not this page's id, which follows the requested `bangla_openbookqa` spelling.
Four-way multiple-choice question answering in Bangla (answer labels A-D); no fact or passage is supplied at inference time, so the model must supply both the relevant science fact and the connecting common-knowledge step itself. The reference lm-evaluation-harness task scores it zero- or few-shot by comparing the log-likelihood the model assigns to each of the four options.
No model card in ModelSpec reports this benchmark yet.