Ruin Names (BIG-bench)

448 four-way questions: pick the funny one-character edit of a movie or band name; BIG-bench Hard keeps 250.

Also known as: Ruin a Name with One Edit, BBH ruin_names

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryreasoning
SubcategoryBIG-bench (and BBH) humorous one-edit movie and band names
Page statusunknown
Metricmultiple_choice_grade
Directionhigher_is_better
Unit%
Dataset size448
Dataset licenceApache-2.0
PublisherGoogle (BIG-bench collaboration)

What it measures

ruin_names shows a movie or artist name and four near-edits. One edit is a humorous ruin from Twitter hashtags #RuinAFilmTitle and #RuinABandName; the others are non-funny Levenshtein-1 distractors. Authors Ryan Teehan, Vinay Prabhu, and Nick Roberts. The skill is spotting the joke, not generating names. BIG-bench Hard keeps a 250-item sample.

Task format

Multiple choice with preferred metric multiple_choice_grade. append_choices_to_input is true. 447 items have four options; one item has three. Dummy-model header: 448 multiple-choice queries. Canary GUID embedded.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub