CIFAR-10 Classification (BIG-bench encodings)

A 20,000-item BIG-bench task that classifies CIFAR-10 test images from base64 PNG strings or hex pixel arrays, without a vision encoder.

Also known as: CIFAR-10 Test, BIG-bench cifar10_classification

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categorymultimodal
SubcategoryBIG-bench 10-way CIFAR-10 test classification from base64 PNG or hex pixels
Page statusunknown
Metricmultiple_choice_grade
Directionhigher_is_better
Unit%
Dataset size20000
Dataset licenceApache-2.0
PublisherGoogle (BIG-bench collaboration); task author at CMU

What it measures

cifar10_classification asks a language model to name which of ten object classes is shown in a CIFAR-10 test image, but the image is not pixels in a vision tower: it is either a base64 PNG string (RFC 3548) or a hex dump of the 32x32 RGB grid. Peiyuan Liao built two subtasks (base64 and hex) over the same test images. The probe is whether a text model can decode those encodings and still classify, not standard CIFAR-10 vision accuracy.

Task format

Ten-way multiple choice (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck). Preferred metric multiple_choice_grade. append_choices_to_input is true. Canary GUID embedded.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub