Cycled Letters (BIG-bench)

BIG-bench programmatic task: rotate a filtered English word and require the original spelling by exact string match.

Also known as: cycled letters

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryreasoning
SubcategoryBIG-bench programmatic word uncycling (exact string match)
Page statusunknown
Metricexact_str_match
Directionhigher_is_better
Unit%
Dataset size20975
Dataset licenceApache-2.0
PublisherGoogle (BIG-bench collaboration)

What it measures

cycled_letters rotates an English word and asks the model to print the original. Example from the task README: input imitedl, target limited. The word list is the intersection of NLTK Brown and NLTK words, length greater than one, with rotations that would be valid for two targets removed. Dummy-model header: 100 free-text queries. Not in [BIG-Bench Hard](bbh.md).

Task format

Programmatic BIG-bench task. Default 100 trials, seed 42. Preamble "Please unscramble the letters into a word Q: {} A: ". Preferred metric exact_str_match. Canary GUID embedded. Zero-shot in evaluate_model (number_of_shots=0).

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub