{
 "body": "\n## What it measures\n\njfinqa tests multi-step numerical reasoning over real Japanese corporate financial statements, drawn from\nactual filings on EDINET, the disclosure system run by Japan's Financial Services Agency, and spanning\nJ-GAAP, IFRS and US-GAAP accounting standards. Questions require one to six chained arithmetic steps --\ngrowth-rate and margin calculations, cross-statement ratio analysis, six-step DuPont decomposition -- over\na mix of narrative text before and after a financial table, the same evidence shape FinQA (in this\nrepository) uses for English S&P 500 filings. Three subtasks sit inside the one benchmark and are scored\nseparately: Numerical Reasoning asks for a calculated figure, Consistency Checking asks whether reported\nnumbers agree with each other, and Temporal Reasoning asks for the direction of a year-over-year change.\n\n## How it is scored\n\nGiven pre-table text, a financial table and post-table text from one filing, plus a question, the model\ngenerates a short free-text answer zero-shot, with no examples shown. Scoring runs two ways at once: exact\nstring match, and numerical match with a 1% tolerance for numeric answers, both reported overall and\nper subtask. The maintainer's baseline runs additionally report two reasoning regimes on the same\nquestions -- one with a model's extended-thinking behaviour switched off, one with each provider's default\nmoderate reasoning budget switched on -- because the effect of enabling reasoning on this benchmark varies\nby model and is sometimes negative.\n\n## Dataset and licence\n\n1,000 questions under an Apache-2.0 licence, split 550/200/250 across the three subtasks, confirmed\ndirectly against the released Hugging Face file. Two primary sources disagree on how many companies the\nquestions are drawn from: the benchmark's own GitHub README states 104, following what it describes as an\nApril 2026 expansion and data-mapping fix, while the lm-evaluation-harness README and the Hugging Face\ndataset card's own summary text both state 68. This page could not resolve which figure matches the\ncurrently published file and reports both rather than choose one. Each question carries its source\ncompany, EDINET filing id, filing year and accounting standard as metadata alongside the question, answer\nand gold arithmetic program.\n\n## Who publishes it\n\njfinqa is an independent project by Saichi Ogawa, distributed simultaneously as a GitHub repository, a PyPI\npackage (`jfinqa`) and a Hugging Face dataset under the account ajtgjmdjp. The GitHub repository's own\ncreation date is February 2026, which sits oddly against the project's citation snippet, which lists 2025\nas the year -- this page records the citation's stated year in the `paper.year` field but notes the\nrepository-metadata date here as the more directly observable signal of when the project actually\nappeared.\n\n## Lineage\n\nNo predecessor or successor is tracked for this id in this repository, but jfinqa is explicit about its own\ndesign lineage: its README states it uses \"the same data format as FinQA\" (Chen et al.) specifically for\ncross-benchmark comparison. That makes it a close relative of `fin_qa` (in this repository) in format only\n-- jfinqa is an independently collected set of Japanese EDINET filings, not a translation of FinQA's\nEnglish S&P 500 questions. It is a different project again from `financebench` (open-book QA over US SEC\nfilings), `finbench` (tabular credit-risk classification) and `buysidefinbench` (bilingual\nChinese/English equity-research multiple-choice), the other financial-domain pages in this repository:\njfinqa is the only one of the five built specifically around Japanese-language, EDINET-sourced, multi-step\nnumerical calculation.\n\n## Saturation and contamination\n\nThe maintainer's own baseline table, undated but bracketed between the dataset's April 2026 update and the\nrepository's August 2026 last push, puts gpt-5.4-mini at 93.7% overall accuracy under a no-reasoning\nsetting, ahead of the larger \"frontier\" gpt-5.4 at 90.6-91.9%. The same README states that Temporal\nReasoning sits at or above 98% for the top seven models tested and Consistency Checking is similarly\nnear-ceiling, while Numerical Reasoning still spans roughly 80-90% among top models -- its own words\ndescribe Numerical Reasoning as now the subtask that actually discriminates between strong models. An\nearlier \"pre-audit\" baseline table for a prior version of the dataset is marked explicitly as not\ncomparable to current numbers. Contamination risk is medium: the dataset is only a few months old as of\nthis research, which limits exposure so far, but every question's exact answer and gold arithmetic program\nare published alongside the question itself, functioning as a full answer key rather than a held-out set.\n\n## How to run it\n\nThe lm-evaluation-harness group `jfinqa` runs all three subtask tasks (`jfinqa_numerical`,\n`jfinqa_consistency`, `jfinqa_temporal`) and reports a size-weighted combined exact_match and\nnumerical_match, per the group's own `aggregate_metric_list`. Generation is zero-shot, greedy, capped at\n256 tokens per answer. The benchmark is also runnable independently of any harness through its own PyPI\npackage, which exposes a `load_dataset` / `evaluate` API and a CLI, and it has been integrated into\nllm-jp-eval as a separate contribution. Because the maintainer's own published baselines mix an\nundocumented current run against an explicitly deprecated \"pre-audit\" run on an earlier dataset version,\nconfirm which dataset revision a reported score used before comparing it against another source.\n\n## Reading the numbers\n\nA high jfinqa score shows a model can extract the right figures from a Japanese financial table and chain\nseveral arithmetic steps correctly, including across different accounting standards. Consistency Checking\nand Temporal Reasoning are close to solved among current top models, so a high score on those two subtasks\nmainly confirms competence rather than distinguishing models; Numerical Reasoning is where the remaining\ngap sits, and is the subtask worth weighing most heavily when comparing models on this benchmark. Because\nenabling a model's extended-reasoning mode helps some models and hurts others here, compare scores from the\nsame reasoning regime rather than across regimes, and check the dataset revision behind any score given the\nunresolved company-count discrepancy and the existence of a deprecated earlier baseline table.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "jfinqa",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "JFinQA",
   "jfinqa: Japanese Financial Numerical Reasoning QA Benchmark"
  ],
  "category": "domain",
  "contamination": {
   "note": "The dataset was created in February 2026 and last updated in April 2026, only a few months before this research, which limits how much current model training could already include it. Against that, every question's exact answer and gold arithmetic program are public in the same Hugging Face file as the question and source table, functioning as a full answer key rather than a held-out set.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "ja"
   ],
   "license": "Apache-2.0",
   "modalities": [
    "text",
    "tabular"
   ],
   "public_test_set": true,
   "size": 1000,
   "size_note": "1,000 questions, confirmed directly against the released dataset (Hugging Face datasets-server): 550 Numerical Reasoning, 200 Consistency Checking, 250 Temporal Reasoning, matching the harness's own per-subtask task split exactly. Two primary sources disagree on the number of source companies: the benchmark's own GitHub README states 104 companies following an April 2026 \"EDINET-mapping\" update and an expansion from an earlier version, while both the lm-evaluation-harness README and the Hugging Face dataset card's own description text state 68 companies. This page could not establish from the sources read which figure the currently published 1,000-question file matches, and reports both readings rather than pick one.\n",
   "splits": "single 'test' split under four Hugging Face configs: 'all' (1,000 rows) plus one config per subtask (numerical_reasoning 550, consistency_checking 200, temporal_reasoning 250)",
   "url": "https://huggingface.co/datasets/ajtgjmdjp/jfinqa"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 6, slice D",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "jfinqa",
   "opencompass": "",
   "other": "Also integrated into llm-jp-eval according to the project's own README (credited as PR #230 there), and distributed as a standalone Python package (`pip install jfinqa`) with its own CLI and library API usable independently of any harness.\n"
  },
  "id": "jfinqa",
  "last_updated": "2026-08",
  "leaderboard_url": "https://ajtgjmdjp.github.io/jfinqa-leaderboard/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "jfinqa tests whether a model can perform multi-step arithmetic over Japanese corporate financial statement tables pulled from real EDINET filings (the disclosure system run by Japan's Financial Services Agency), spanning J-GAAP, IFRS and US-GAAP accounting standards. It is not a classification or simple-lookup task: questions require one to six chained arithmetic steps -- growth-rate and margin calculations, ratio analysis, DuPont decomposition -- over a mix of pre-table text, a financial table, and post-table text, the same evidence shape FinQA (in this repository) uses for English SEC filings. Three separately-scored subtasks sit inside the one benchmark: Numerical Reasoning (calculate a financial metric), Consistency Checking (verify that reported figures agree with each other) and Temporal Reasoning (determine the direction of a year-over-year change).\n",
  "metric": {
   "baseline_note": "No random or human baseline is published. The maintainer's own baseline runs report two reasoning \"regimes\" on the same 1,000 questions -- R0 (model thinking/reasoning disabled) and R1 (each provider's default moderate reasoning budget) -- specifically to separate how much of a model's score depends on whether extended reasoning is switched on, since the effect turns out to differ by model and is sometimes negative.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "exact_match and numerical_match (1% tolerance on numeric answers), reported overall and per subtask",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "jfinqa",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "jfinqa: Japanese Financial Numerical Reasoning QA Benchmark",
   "url": "https://github.com/ajtgjmdjp/jfinqa",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Saichi Ogawa"
   ],
   "org": "Independent project (GitHub user ajtgjmdjp); distributed simultaneously as a GitHub repository, a PyPI package and a Hugging Face dataset",
   "url": "https://github.com/ajtgjmdjp/jfinqa"
  },
  "released": "2026-02",
  "repo_url": "https://github.com/ajtgjmdjp/jfinqa",
  "saturation": {
   "as_of": "",
   "note": "The maintainer's own baseline table (undated, bracketed between the dataset's April 2026 Hugging Face update and the repository's August 2026 last push) puts gpt-5.4-mini at 93.7% overall accuracy under its \"R0\" no-reasoning regime, ahead of the larger gpt-5.4 \"frontier\" model at 90.6-91.9% depending on regime. The README's own analysis states Temporal Reasoning is at or above 98% for the top seven models and Consistency Checking is similarly near-ceiling, while Numerical Reasoning still spans 80.4-89.5% among top models and is described there as \"now the discriminating subtask\" -- matching this repository's `watch` status (spread has collapsed on two of three subtasks, one still separates models). An earlier \"pre-audit\" baseline table exists in the same README for a prior version of the dataset and is explicitly marked not comparable to the current numbers.\n",
   "status": "watch",
   "top_score": 93.7
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "jfinqa task README, lm-evaluation-harness",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/jfinqa/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "_jfinqa.yaml: group definition and weighted aggregate_metric_list",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/jfinqa/_jfinqa.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "jfinqa_numerical.yaml: zero-shot generation config and metrics",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/jfinqa/jfinqa_numerical.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "jfinqa GitHub README: dataset statistics, subtasks, baseline results, FinQA-compatibility note",
    "url": "https://raw.githubusercontent.com/ajtgjmdjp/jfinqa/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "ajtgjmdjp/jfinqa repository metadata (licence, creation and push dates), GitHub API",
    "url": "https://api.github.com/repos/ajtgjmdjp/jfinqa"
   },
   {
    "accessed": "2026-09-08",
    "title": "ajtgjmdjp/jfinqa dataset metadata, Hugging Face API",
    "url": "https://huggingface.co/api/datasets/ajtgjmdjp/jfinqa"
   },
   {
    "accessed": "2026-09-08",
    "title": "ajtgjmdjp/jfinqa split and feature info, datasets-server",
    "url": "https://datasets-server.huggingface.co/info?dataset=ajtgjmdjp/jfinqa"
   }
  ],
  "status": "active",
  "subcategory": "numerical reasoning over Japanese corporate financial statements (EDINET filings): calculation, internal-consistency and year-over-year trend questions",
  "summary": "jfinqa tests multi-step numerical reasoning over real Japanese corporate financial statements from EDINET filings, across three subtasks: calculation, internal-consistency checking and trend direction.",
  "tags": [
   "domain",
   "financial",
   "japanese",
   "numerical-reasoning",
   "tabular",
   "edinet"
  ],
  "task_format": "Given pre-table text, a financial table and post-table text drawn from one company's EDINET filing, plus a question, the model generates a short free-text answer zero-shot -- a figure, a percentage, or a yes/no-style judgement depending on the subtask.\n"
 }
}