Rhyming (BIG-bench)

Two BIG-bench English subtasks: 680 five-way rhyme picks from CMUdict, and 273 verse rhyme-scheme strings.

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	knowledge
Subcategory	BIG-bench English rhyme identification (680) plus verse rhyme-scheme generation (273)
Page status	unknown
Metric	multiple_choice_grade (rhyme_multiple_choice); exact_str_match (rhyme_scheme)
Direction	higher_is_better
Unit	%
Dataset size	953
Dataset licence	Apache-2.0
Publisher	Google (BIG-bench collaboration)

What it measures

rhyming has two JSON subtasks by Oscar Knagg. rhyme_multiple_choice shows a query word and five candidates; one rhymes under CMUdict phonemes. rhyme_scheme shows a verse and asks for its letter rhyme scheme. Each gold target is a two-string list such as AA and aa. Neither subtask scores meter or performance. Parent task.json holds only metadata.

Task format

Two subtasks. rhyme_multiple_choice: five-way multiple_choice_grade (680 dummy-model multiple-choice queries). rhyme_scheme: exact_str_match free response (273 dummy-model free-text queries). Canary GUID embedded in the parent and in each subtask file.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub