Unnatural In-Context Learning

Synthetic identity, date, reversal and arithmetic subtasks test in-context pattern induction outside natural training distributions.

Also known as: BIG-bench Unnatural In-Context Learning

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryreasoning
Subcategoryfew-shot program induction on synthetic formats
Page statusactive
Metricexact completion accuracy
Directionhigher_is_better
Unit%
Dataset size73420
PublisherBIG-bench collaboration

What it measures

Few-shot examples define synthetic transformations; the model must produce the next output. Subtasks cover identity, date formats, reversal and unusual two-digit addition.

Task format

Free-text numerical or symbolic completion with variable numbers of demonstrations.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub