Emoji Movie

A 100-item BIG-bench Lite quiz that asks which popular movie an emoji plot describes, scored mainly by five-way multiple_choice_grade.

Also known as: BIG-bench emoji_movie

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	knowledge
Subcategory	BIG-bench Lite five-way emoji-to-movie quizzes (100 items)
Page status	unknown
Metric	multiple_choice_grade
Direction	higher_is_better
Unit	%
Dataset size	100
Dataset licence	Apache-2.0
Publisher	Google (BIG-bench collaboration)

What it measures

emoji_movie gives an emoji string that sketches a film plot, not the title letters, and asks for the movie. Each item also lists five English titles (one gold, four random IMDb Top 250 decoys) so the task is both generative and five-way multiple choice. Author Zijie J. Wang. English plus emoji. The skill is emoji semantics plus popular-culture plot memory.

Task format

Five-option multiple choice with a free-text target. preferred_score multiple_choice_grade; also rouge, bleu, exact_str_match. append_choices_to_input true. Canary GUID embedded. BIG-bench Lite member.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub