A BIG-bench task that maps lemmas and sentences through tense, plural, questions, negation, and pronouns from a handful of shots.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | BIG-bench few-shot English/German morphological and syntactic mappings |
| Page status | unknown |
| Metric | exact_str_match |
| Direction | higher_is_better |
| Dataset size | 15527 |
| Dataset licence | Apache-2.0 |
| Publisher | Google (BIG-bench collaboration) |
linguistic_mappings asks whether a model can apply a grammatical mapping after a few examples, rather than retrieve a memorised surface form. Morphology covers English and German past tense and English plurals, using CELEX-2 lemmas in Berko-style carrier sentences. Syntax covers English polar questions, main-verb negation, and pronoun or reflexive replacement, from a hand-written CFG. JSON subtasks mix items; programmatic subtasks also cross in-domain versus out-of-domain shots (regularity, number and tense, or name gender) at 0–32 shots (0–16 for plural).
Free-text generation. Preferred metric exact_str_match on every JSON task.json opened here. Parent task.json holds name and keywords only. Canary GUID embedded. Not multiple choice.
No model card in ModelSpec reports this benchmark yet.