The 400-task public evaluation split of the original ARC-AGI (ARC-AGI-1) grid-puzzle benchmark, fully public since 2019 and now superseded for frontier evaluation by ARC-AGI-2.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | abstract visual reasoning (ARC-AGI-1 public evaluation split, 400 tasks) |
| Page status | superseded |
| Metric | % of tasks solved (exact grid match) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 400 |
| Dataset licence | Apache-2.0 |
| Publisher | Originally released independently by François Chollet; now stewarded by the ARC Prize Foundation |
This benchmark is the public evaluation split of ARC-AGI-1 (the original Abstraction and Reasoning Corpus, sometimes just called "ARC" or "ARC-AGI"), François Chollet's 2019 grid-puzzle test of fluid, general reasoning. Each task shows a handful of input/output grid pairs sharing a hidden transformation rule, plus a new input grid; the model must infer the rule from the examples alone and produce the matching output grid. Grids are small matrices of integers 0-9 (shown to humans as colours), and every task is hand-designed to be solvable by most people without specialist knowledge, while resisting brute-force pattern matching. This id names specifically the 400-task public evaluation set -- the "public tasks" split described in the ARC Prize competition's own guide -- not the separate training set or the fully private set used to judge the ARC Prize grand prize.
A handful of paired example grids (input and output) plus one or more test input grids; the model must output the exact matching grid. The original repository's protocol allows 3 attempts per test input; OpenCompass's implementation instead scores a single zero-shot text generation per task (no retries), so scores under that harness are not directly comparable to the original protocol or to ARC-AGI-2's 2-attempts convention.
No model card in ModelSpec reports this benchmark yet.