Geometric Shapes

360 SVG path strings labelled with a shape name; BIG-bench scores multiple_choice_grade, and BBH keeps a 250-item slice.

Also known as: BIG-bench geometric_shapes, BBH geometric_shapes

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryreasoning
Subcategoryname a polygon from an SVG path string (BIG-bench / BBH)
Page statusunknown
Metricmultiple_choice_grade
Directionhigher_is_better
Unit%
Dataset size360
Dataset licenceApache-2.0
PublisherGoogle (BIG-bench collaboration)

What it measures

geometric_shapes shows an SVG path `d` string on a 100×100 canvas and asks which simple shape it draws. Author Zijie J. Wang wrote the paths to test whether a language model can parse move/line commands and name the geometry, without a raster image. task.json has ten gold labels with 36 items each: line, triangle, rectangle, kite, pentagon, hexagon, heptagon, octagon, circle, sector. Trapezoid and ellipse appear only as extra option keys on some items, not as gold classes. English names; the input is path syntax, not pixels. [BIG-Bench Hard](bbh.md) keeps 250 of these items.

Task format

Multiple choice with preferred metric multiple_choice_grade; metrics also list rouge, bleu, and exact_str_match. append_choices_to_input is true. 288 items have 10 options and 72 have 11. Canary GUID embedded. The auto-generated README header reports 359 multiple-choice and 359 free-text dummy queries; hails/bigbench geometric_shapes_zero_shot default is also 359.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub