17,177-item machine translation of PIQA's physical-commonsense-reasoning questions into Bangla, built with an automated Google-Translate-plus-LLM-rewriting pipeline the authors call EST.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | Bangla physical commonsense reasoning, machine-translated from PIQA |
| Page status | active |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 17177 |
| Dataset licence | MIT (Hugging Face dataset card) |
| Publisher | Hishab (Hishab Singapore Pte. Ltd), with the University of Central Florida and the Qatar Computing Research Institute |
Bangla PIQA tests physical commonsense reasoning in Bangla: given a goal stated in a short sentence and two candidate solutions, a model must pick the more physically sensible one, the same task PIQA poses in English. It is a translation rather than a from-scratch Bangla benchmark -- the TituLLMs paper that introduces it states it is "a Bangla translation of the ... PIQA dataset," produced with the authors' own Expressive Semantic Translation (EST) pipeline, which runs a standard machine-translation pass and then refines it through iterative, LLM-generated candidate re-translations, ranked and selected automatically. No human translation or per-item human verification of the finished Bangla items is described for this dataset.
Two-way multiple-choice question answering in Bangla (answer labels A/B): given a goal sentence, the model selects the more physically appropriate of two candidate solutions. The reference lm-evaluation-harness task scores it zero- or few-shot by comparing the log-likelihood the model assigns to each of the two options.
No model card in ModelSpec reports this benchmark yet.