A 342-item BIG-bench task that asks yes or no whether an action in a short moral-dilemma story is permissible, using majority human labels.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | BIG-bench trolley-style yes/no moral dilemmas (342 items) |
| Page status | unknown |
| Metric | multiple_choice_grade |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 342 |
| Dataset licence | Apache-2.0 |
| Publisher | Google (BIG-bench collaboration); authors at Stanford |
moral_permissibility gives a short English story, often a trolley-problem variant, and asks whether an action is morally permissible. The target is the majority human answer from the psychology paper that published that story, binarised Yes/No. Authors Allen Nie and Tobias Gerstenberg present it as a diagnostic of whether models track the same factors humans do (personal force, intent, who benefits), not as a test of "correct" ethics. English text.
Two-option multiple_choice_grade (Yes/No). append_choices_to_input false. Zero-shot in the task keywords. Canary GUID embedded. 342 dummy-model queries. Each example carries a comment with the source paper and, when the authors had it, raw human agreement.
No model card in ModelSpec reports this benchmark yet.