An 80-item BIG-bench task that picks the Russian proverb closest in non-literal meaning to an English proverb.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | BIG-bench English-to-Russian proverb analogy (80 four-way items) |
| Page status | unknown |
| Metric | multiple_choice_grade |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 80 |
| Dataset licence | Apache-2.0 |
| Publisher | Google (BIG-bench collaboration) |
english_russian_proverbs gives an English proverb or idiom and four Russian sayings. The model must choose the Russian item whose non-literal meaning is closest, not the literal gloss. Denis Kleyko and Cameron Diao wrote the 80 four-way items. Distractors were chosen to be unrelated or to share surface nouns and verbs with the query so that word-for-word translation is a trap. Source lists named in the README are Bodrova (2007), Gvarjalaże and Mchedlishvili (1971), and Wikiquote's Russian proverbs page. The skill is cross-lingual figurative analogy, not [flores](flores.md)-style translation and not [english_proverbs](english_proverbs.md).
Four-option multiple choice, preferred metric multiple_choice_grade. task_prefix asks for a Russian proverb/idiom close in meaning. example_input_prefix is "English proverb:"; example_output_prefix is "Russian proverb:". append_choices_to_input is true. Keywords include zero-shot, one-shot, and many-shot. Canary GUID embedded.
No model card in ModelSpec reports this benchmark yet.