Repeat Copy Logic (BIG-bench)

32 English instructions that copy, repeat, and insert words; a BIG-bench Lite exact-string-match task.

Also known as: BIG-bench Lite repeat_copy_logic

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryreasoning
SubcategoryBIG-bench Lite free-response nested copy-and-logic instructions
Page statusunknown
Metricexact_str_match
Directionhigher_is_better
Unit%
Dataset size32
Dataset licenceApache-2.0
PublisherGoogle (BIG-bench collaboration)

What it measures

repeat_copy_logic gives one English instruction that mixes text to copy with counting, every-other, and nested repeats. The model must emit the exact output string. The README credits the BIG-bench organizers; task.json lists contributor nfiedel. It extends the Neural Turing Machines Repeat Copy setup (arXiv:1410.5401) from a scalar copy-count to nested natural language. One of the 24 BIG-bench Lite JSON tasks.

Task format

Free-response JSON. Preferred metric exact_str_match. task_prefix is "repeat with logic:\\n". Dummy-model header: 32 free-text queries, 0 multiple choice. Canary GUID embedded.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub