Alignment of Simplicity Priors for Turing-Complete Concept Learning

A 6,390-query BIG-bench task that tests few-shot learning of 426 P3 binary-string programs from machine-teaching witness sets.

Also known as: simp_turing_concept, Simplicity priors for Turing-complete concept learning

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryreasoning
SubcategoryBIG-bench few-shot P3 binary-string programs from machine-teaching witness sets (6,390 queries)
Page statusunknown
Metricexact_str_match
Directionhigher_is_better
Unit%
Dataset size6390
Dataset licenceApache-2.0
PublisherGoogle (BIG-bench collaboration); Universitat Politècnica de València

What it measures

simp_turing_concept shows Input/Output pairs of binary strings and asks for the next output, testing whether a model shares a simplicity prior with a machine teacher for P3 (a small Turing-complete language). 426 concepts; three nested teaching batches (witness set, AS I, AS II); five held-out test strings per concept (max length 5). Parent task.json has no examples; subtasks do. Dummy-model header: 0 multiple-choice and 6,390 free-text queries. Witness-set JSON counted 2,130 examples (426 × 5). Preferred metric exact_str_match. Must be run as BIG-bench zero-shot because shots are already in the prompt. Canary GUID embedded. Not in BIG-Bench Hard.

Task format

Free-text continuation. Prompt style Input: … Output: with predetermined few-shot pairs. output_regex "('.*?')". Keywords many-shot, logical reasoning, computer code, json, free response. Subtasks witness_set, additional_set_1, additional_set_2.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub