Conlang Translation

A 164-item BIG-bench Lite task that infers a ciphered language from a few examples, then translates to or from English.

Also known as: BIG-bench conlang_translation, Conlang Translation Problems

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	translation
Subcategory	BIG-bench Lite few-shot ciphered-language translation (164 items, 16 direction pairs)
Page status	unknown
Metric	rougeLsum
Direction	higher_is_better
Dataset size	164
Dataset licence	Apache-2.0
Publisher	Google (BIG-bench collaboration)

What it measures

conlang_translation gives a short grammar-puzzle prompt in the style of NACLO and IOL problems: several sentence pairs between English and a constructed or ciphered language, then a held-out sentence to translate. Real words are replaced; the model must induce lexicon, morphology, and word order from the examples. Eight language pairs, each in both directions. Authors: Rowan Jacobs, James Koppel, and Mimee Xu. Not conventional corpus MT and not [LingOly](lingoly.md).

Task format

Free-text generation. Preferred metric rougeLsum; also bleu, exact_str_match, and other ROUGE variants. Parent task.json holds name and keywords; items live in 16 subtask JSON files. Canary GUID embedded. BIG-bench Lite member.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub