8,281 expert-written QA pairs over S&P 500 earnings-report excerpts, scored by checking a generated arithmetic reasoning program rather than just a final number.
unassessed
| Category | domain |
|---|---|
| Subcategory | numerical reasoning over financial reports (program generation) |
| Page status | active |
| Metric | program accuracy and execution accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 8281 |
| Dataset licence | Stated two ways. The GitHub repository's own LICENSE file, at the repository root, is MIT (no separate licence is carved out for the JSON data files versus the code). The paper itself, however, describes the released question/program annotations as "Enhanced Data" built on top of FinTabNet (Zheng et al., 2021), which is distributed under CDLA-Permissive-1.0, and states that this licence is specifically what permits FinQA to publish its annotations over that underlying data. This page records both readings rather than picking one. |
| Publisher | University of California, Santa Barbara, with J.P. Morgan, Pennsylvania State University and Carnegie Mellon University |
FinQA gives a model an excerpt from a real company's earnings report -- the text before a table, the table itself, and the text after it -- together with a question a financial analyst might ask (for example, "what was the percentage change in net sales from 2005 to 2006?"). Instead of asking for a bare numeric answer, the task requires the model to produce an executable reasoning program in a small domain-specific language: six arithmetic operations (add, subtract, multiply, divide, exp, greater) plus four table-aggregation operations (table-sum, table-average, table-max, table-min), chained so that later steps can reference earlier results. It is single-turn, English-language, and mixes unstructured text with a structured table in the same input.
Given pre-table text, a table (row and column headers plus values), post-table text and a question, generate a sequence of DSL operation tokens, e.g. `divide(9413, 20.01), divide(8249, 9.48), subtract(#0, #1)`, that when executed produces the answer. Scored by comparing the generated program to a human-written gold program (program accuracy) and separately by executing the generated program against the input table and checking the resulting number or yes/no value (execution accuracy).
No model card in ModelSpec reports this benchmark yet.