A 2,000-item BIG-bench Lite task that induces a constructed language from paired sentences and translates one more sentence either way.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | BIG-bench Lite Rosetta-stone puzzles over constructed languages (2,000 items) |
| Page status | unknown |
| Metric | exact_str_match |
| Direction | higher_is_better |
| Dataset size | 2000 |
| Dataset licence | Apache-2.0 |
| Publisher | Google (BIG-bench collaboration) |
linguistics_puzzles is a Rosetta-stone linguistics-olympiad item written over a fresh constructed language, not a natural low-resource language. Each example shows several English/conlang sentence pairs, then asks for one more translation either into English or into the conlang. Nathan A. Chi generated a different conlang per item (case, number, headedness, and similar features) so web lookup of a real language should not help. It is not [LingOly](lingoly.md) (real olympiad sheets) and not [conlang_translation](conlang_translation.md) (ciphered real languages on BIG-bench Lite).
Free-text generation. Preferred metric exact_str_match; also bleu and rouge. One parent task.json holds all 2,000 examples. Canary GUID embedded. BIG-bench Lite member.
No model card in ModelSpec reports this benchmark yet.