CharXiv Reasoning

The reasoning half of CharXiv: one open-ended question per chart that requires combining several visual elements, not just reading a labelled value.

Also known as: CharXiv-R

unassessed

What it measures

CharXiv Reasoning is the reasoning-question half of the CharXiv family: for each of 2,323 real charts pulled from arXiv papers, the model must answer one open-ended question that requires synthesising information across multiple parts of the chart -- comparing series, computing a derived value, cross-referencing the legend against the plotted data -- rather than reading a single labelled point. The benchmark's own taxonomy further splits reasoning questions into four answer-type categories: text-in-chart, text-in-general, number-in-chart and number-in-general, depending on whether the answer is text or a number and whether it is read directly off the chart or requires outside synthesis.

Task format

Open-ended, free-text short-answer question about one chart image, one per chart, not multiple-choice. Models typically reason before committing to a final answer; there is no fixed answer template beyond what individual harnesses impose.

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub