80-item BIG-bench binary task: decide whether a short English text is intended as a dark joke.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | BIG-bench binary dark-joke vs not-a-joke classification |
| Page status | unknown |
| Metric | multiple_choice_grade |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 80 |
| Dataset licence | CC0-1.0 (task README); Apache-2.0 (google/BIG-bench repository) |
| Publisher | Saarland University / DFKI (via the BIG-bench collaboration) |
dark_humor_detection shows one English snippet and asks whether it is intended as a dark joke. Gold labels are joke versus not a joke, 40 each in task.json. Non-jokes are punchlines rewritten to similar length. The authors (Saarland/DFKI) say the target is intent, not funniness. Dummy-model header: 80 multiple-choice queries. Not in [BIG-Bench Hard](bbh.md).
JSON multiple-choice, preferred_score multiple_choice_grade. task_prefix asks for "joke" or "not a joke". append_choices_to_input is false. Canary GUID embedded. lm-eval ships bigbench_dark_humor_detection_multiple_choice and bigbench_dark_humor_detection_generate_until on hails/bigbench, dataset_name dark_humor_detection_zero_shot.
No model card in ModelSpec reports this benchmark yet.