{
 "body": "\n## What it measures\n\nFinBench tests whether a model can predict financial risk \u2014 will this customer default on a credit\ncard or loan, commit credit-card fraud, or churn \u2014 from a record of that customer. The starting\npoint is a row of tabular data: age, income, education, credit score, loan type and similar fields.\nFinPT, the method the benchmark was built to evaluate, first prompts a language model to turn that\nrow into a fluent natural-language \"customer profile\" paragraph, then fine-tunes a foundation model\non the profile text, with a small classification head, to predict a binary risky/not-risky label.\n\nIt is ten separate binary-classification datasets, not one: two on credit-card default, three on\nloan default, two on credit-card fraud, and three on customer churn, each sourced from a different\npublic Kaggle dataset and unified into one loading interface. Every dataset is skewed, with the\npositive (risky) class ranging from well under 1% to just over a quarter of instances, so the task\nalso tests whether a model handles severe class imbalance rather than just predicting the majority.\n\n## How it is scored\n\nEvery dataset is scored with F1 on the positive (risky) class, not accuracy \u2014 the authors state\nplainly that accuracy would let a model do well by defaulting to the majority \"not risky\" label\ngiven how imbalanced these datasets are. A single \"FinBench\" number, where one is reported, is an\nunweighted average of the ten per-dataset F1 scores. The original paper's own protocol fine-tunes a\nfoundation model (with all parameters, or just the last decoder block for very large models) on the\nLLM-generated profile text, training four times with different random seeds and averaging; a\nseparate experiment shows that zero-shot or few-shot prompting of the same models, without\nfine-tuning, scores under 10 F1 on average. That gap means a \"FinBench\" score's method \u2014 fine-tuned\nclassifier versus prompted judgment \u2014 needs to be known before treating two scores as comparable.\n\n## Dataset and licence\n\nThe ten datasets were screened from \"hundreds\" of Kaggle datasets for quality, popularity and\ncolumn meaningfulness, then unified into one structure with roughly 333,000 labeled instances in\ntotal. Each dataset ships its own train, validation and test split: a fixed 30% test set, with the\nremaining 70% split 9:1 into train and validation. Alongside the raw tabular rows, the dataset\nprovides an instruction template and the LLM-generated profile text for each instance, plus\nper-table statistics for algorithms that need them. The Hugging Face dataset card lists the licence\nas CC BY-NC 4.0; the code repository that builds and loads it is released separately under MIT.\n\n## Who publishes it\n\nFinBench was introduced alongside the FinPT method by Yuwei Yin, Yazheng Yang and Qi Liu at the\nUniversity of Hong Kong's Department of Computer Science, with Jian Yang at DAMO Academy, Alibaba\nGroup, in the paper \"FinPT: Financial Risk Prediction with Profile Tuning on Pretrained Foundation\nModels,\" posted to arXiv in July 2023. The paper is marked as a University of Hong Kong preprint;\nno separate conference or journal acceptance was found in the sources read. The authors maintain\nthe reference code at github.com/YuweiYin/FinPT and the dataset on Hugging Face.\n\n## Lineage\n\nNo predecessor or successor benchmark, and no variant or subset page, was established for FinBench\nin this repository's own materials; this appears to be the only page for this benchmark family here.\nThe name is easy to confuse with other finance-benchmark efforts that use similar wording \u2014 notably\nFinBen, a much larger 36-dataset, 24-task holistic financial benchmark, and FinanceBench, a\n150-question open-book financial-filings QA set \u2014 but those are distinct projects with their own\npapers and datasets, not variants of this one. Readers encountering a \"finbench\" score should\nconfirm which of these it actually refers to before comparing it across sources.\n\n## Saturation and contamination\n\nThe paper's own baselines leave clear room to improve: tree-based models average 44-47 F1, small\ntabular neural networks average 24-40, and the strongest FinPT configuration (Flan-T5-Base, fully\nfine-tuned) averages 49.17 \u2014 under half of the maximum. Those numbers all come from fine-tuning\nopen, small-to-mid-size models on the ten datasets; this repository found no current, independent\ntracker scoring today's frontier chat models against FinBench under one stated protocol, so a\npresent-day saturation call is not established here. Contamination risk is high: all ten source\ntables are public, long-circulating Kaggle datasets \u2014 several, like the Taiwan credit-card-default\nset, are classic tabular-ML datasets that predate this benchmark by years \u2014 and labels sit locally\nwith the data rather than behind a held-out server, so a model trained on general web data has a\nreal chance of having seen the underlying tables even without seeing FinPT's specific profile text.\n\n## How to run it\n\nThe authors' own repository (YuweiYin/FinPT) is the reference implementation: it constructs\ncustomer-profile text via the OpenAI API from the raw tabular rows, then fine-tunes a foundation\nmodel \u2014 the paper tests BERT, FinBERT, GPT-2, T5, Flan-T5 and LLaMA at various sizes \u2014 with a small\nfeed-forward classifier on its hidden states, training four times per configuration with different\nseeds and reporting the average. No standard harness (lm-evaluation-harness, inspect_evals, HELM,\nOpenCompass, BIG-bench) was confirmed to carry FinBench as a task. Because the original protocol is\nfine-tuning, not prompting, any zero-shot or few-shot FinBench number for a chat model reflects a\ndifferent evaluation method than the one the benchmark's own paper validates, and it is not\nestablished here which method any given reported score used.\n\n## Reading the numbers\n\nA high FinBench score means a model, after seeing the profile text, separates risky from\nnon-risky customers well on ten specific, imbalanced Kaggle datasets \u2014 it says nothing about\nfinancial reasoning in general, or about how the model would do on a bank's actual, differently\ndistributed customer base. Because F1 rather than accuracy is the metric, small changes in a score\ncan reflect the fixed class imbalance rather than genuine skill differences between models. Given\nhow much the original paper's fine-tuned baselines vary by dataset (F1 on the rarest-fraud dataset,\ncf1, tops out near 22 even for the best model), a single averaged number can hide a model that does\nwell on the easier datasets and badly on the hardest one; check the per-dataset breakdown, and the\nevaluation method behind the number, before treating a FinBench score as a settled comparison.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "finbench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4",
   "model_id": "anthropic/claude-opus-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 71.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.6",
   "model_id": "anthropic/claude-opus-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 71.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, anthropic-system-card-mythos"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro",
   "model_id": "google/gemini-2-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 69.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 05-06",
   "model_id": "google/gemini-2-5-pro-preview-05-06",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 69.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 06-05",
   "model_id": "google/gemini-2-5-pro-preview-06-05",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 69.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview TTS",
   "model_id": "google/gemini-2-5-pro-preview-tts",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 69.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o",
   "model_id": "openai/gpt-4o",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 68.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-05-13)",
   "model_id": "openai/gpt-4o-2024-05-13",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 68.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-08-06)",
   "model_id": "openai/gpt-4o-2024-08-06",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 68.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-11-20)",
   "model_id": "openai/gpt-4o-2024-11-20",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 68.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o mini",
   "model_id": "openai/gpt-4o-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 68.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 70B",
   "model_id": "meta/llama-3-1-70b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 55.2,
   "source": "lmarena.ai, provider-reports, safety-evals, domain-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 70B Instruct",
   "model_id": "meta/llama-3-1-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 55.2,
   "source": "lmarena.ai, provider-reports, safety-evals, domain-evals, open-llm-leaderboard-v2"
  }
 ],
 "page": {
  "aliases": [
   "FinBench (FinPT)"
  ],
  "category": "domain",
  "contamination": {
   "note": "All ten source datasets are public Kaggle datasets, several of them long-standing, widely reused tabular-ML datasets (for example the Taiwan credit-card default set and the HMEQ home equity set) that predate FinBench by years and have been used in countless tutorials and papers. Labels are shipped locally with the data rather than held out behind a submission server, and the natural-language \"profile\" text FinPT generates from each row is itself published on Hugging Face. A model trained on general web and code data has a real chance of having seen the underlying tables, even if not the exact profile-text rendering.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC BY-NC 4.0 (dataset, per the Hugging Face dataset card); the accompanying code repository is released separately under MIT.",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 10,
   "size_note": "Ten binary-classification datasets in four sub-categories: credit-card default (cd1, cd2), loan default (ld1, ld2, ld3), credit-card fraud (cf1, cf2) and customer churn (cc1, cc2, cc3), all collected from Kaggle. About 333,000 labeled instances in total, 9 to 120 features per dataset. Each dataset has its own train/validation/test split: test is a fixed 30% of instances, and the remaining 70% splits 9:1 into train and validation.\n",
   "splits": "per-dataset train/validation/test (70/none-extra/30, with train:validation at 9:1)",
   "url": "https://huggingface.co/datasets/yuweiyin/FinBench"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 1b, slice P",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "No standard automated harness (lm-evaluation-harness, inspect_evals, HELM, OpenCompass, BIG-bench) was confirmed to include FinBench. The authors' own repository (YuweiYin/FinPT) is the reference implementation: it builds the customer-profile text via the OpenAI API, then fine-tunes a foundation model with a small classification head on top of its hidden states, training four times per configuration with different seeds and averaging. This differs substantially from a zero-shot or few-shot prompting evaluation of a chat model, so a \"FinBench\" number's method (fine-tuned classifier vs. prompted judgment, and which of the ten datasets were averaged) should be checked before comparing two reported scores.\n"
  },
  "id": "finbench",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "FinBench tests whether a model can predict financial risk \u2014 credit-card default, loan default, credit-card fraud, or customer churn \u2014 from a customer's record. The record starts as a row of tabular data (age, income, credit score, loan type, and so on); FinPT, the method the benchmark was built to evaluate, first has a language model turn that row into a natural-language \"customer profile,\" then fine-tunes a foundation model on the profile text to predict a binary label (risky or not). It is a single-turn, text-only, English-language classification task over ten separate datasets, each drawn from a different Kaggle source and grouped into three risk categories.\n",
  "metric": {
   "baseline_note": "The original paper scores every dataset with F1 rather than accuracy specifically because all ten datasets are class-imbalanced (positive/risky rate ranges from 0.67% to 27.8% across datasets and splits); the authors state plain accuracy would let a model score well by predicting the majority class. No random or human baseline is reported. A single \"FinBench\" number, when one is reported, is an unweighted average of the ten per-dataset F1 scores; the paper's own best average (FinPT on Flan-T5-Base, full fine-tuning) is 49.17.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "F1-score (binary, positive class)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "FinBench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2308.00065",
   "title": "FinPT: Financial Risk Prediction with Profile Tuning on Pretrained Foundation Models",
   "url": "https://arxiv.org/abs/2308.00065",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Yuwei Yin",
    "Yazheng Yang",
    "Jian Yang",
    "Qi Liu"
   ],
   "org": "Department of Computer Science, University of Hong Kong, with DAMO Academy, Alibaba Group",
   "url": "https://github.com/YuweiYin/FinPT"
  },
  "released": "2023-07",
  "repo_url": "https://github.com/YuweiYin/FinPT",
  "saturation": {
   "as_of": "",
   "note": "The paper's own baselines leave real headroom: tree-based models (RandomForest, XGBoost, CatBoost, LightGBM) average 44-47 F1, small tabular neural networks (DeepFM, STG, VIME, TabNet) average 24-40, and the best FinPT configuration (Flan-T5-Base, fully fine-tuned on profile text) averages 49.17 \u2014 well short of the ceiling. Those runs all fine-tune small-to-mid-size open models (up to 13B parameters) on the profile text; the authors separately report that prompting LLaMA and Flan-T5 zero/few-shot, without fine-tuning, scores under 10 F1. This repository found no independent, current tracker that scores today's frontier chat models against FinBench under a stated, comparable protocol, so a present-day saturation read is not established here.\n",
   "status": "open",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "FinPT: Financial Risk Prediction with Profile Tuning on Pretrained Foundation Models",
    "url": "https://arxiv.org/abs/2308.00065"
   },
   {
    "accessed": "2026-09-08",
    "title": "YuweiYin/FinPT (GitHub repository, README and LICENSE)",
    "url": "https://github.com/YuweiYin/FinPT"
   },
   {
    "accessed": "2026-09-08",
    "title": "yuweiyin/FinBench dataset card (Hugging Face)",
    "url": "https://huggingface.co/datasets/yuweiyin/FinBench"
   }
  ],
  "status": "active",
  "subcategory": "financial risk classification",
  "summary": "Ten Kaggle-sourced tabular datasets, turned into natural-language customer profiles, that test whether a model flags credit default, fraud or customer-churn risk.",
  "tags": [
   "finance",
   "tabular-to-text",
   "classification",
   "imbalanced",
   "risk-prediction"
  ],
  "task_format": "Binary classification: given a natural-language customer profile (or the underlying tabular row), predict whether the customer is financially risky (1) or not (0), separately for each of ten datasets across default, fraud and churn.\n"
 }
}