{
 "body": "\n## What it measures\n\nFinancial PhraseBank is three-class sentiment on a single English sentence\nfrom financial news. Annotators at Aalto judged the sentence as an investor\nwould: does it point to a positive, negative, or neutral effect on the\ncompany's value. Everyday sentiment that is not financially relevant is\nneutral. The corpus is a random sample of sentences from LexisNexis news\nabout OMX Helsinki-listed firms, filtered to sentences that contain at\nleast one finance-lexicon entity.\n\nHELM uses that corpus as a generation-of-label classification scenario.\nIt is not open-book filing QA ([financebench](financebench.md),\n[fin_qa](fin_qa.md)) and not a finance knowledge exam\n([financeiq](financeiq.md)).\n\n## How it is scored\n\nThe HELM scenario metadata names `classification_weighted_f1` as the main\nmetric on the test split. The enterprise run spec generates a short Label\n(max 30 tokens) and adds exact-match plus weighted F1/precision/recall over\nthe three label strings. Default agreement is 50, meaning the file of\nsentences where at least half the annotators agreed. Changing `agreement`\nto 66, 75, or 100 changes both the item set and the gold labels' strictness.\nA classifier accuracy from a fine-tune on all-agree is not a HELM number.\n\n## Dataset and licence\n\nThe Hub card licences the work CC-BY-NC-SA-3.0 and asks for a separate\ncommercial licence from Malo or Sinha. Sixteen people annotated; three were\nresearchers and thirteen were Aalto master's students. Each sentence has\nfive to eight overlapping labels. Four public gold standards keep sentences\nwith 100%, \u226575%, \u226566%, or \u226550% agreement.\n\nThe paper, Hub summary, and HELM docstring say 4,840 sentences. The Hub\n`sentences_50agree` config lists 4,846 rows. This page records 4,846 as the\nHub count for the HELM default config and treats 4,840 as the authors'\nstated collection size. There is no publisher train/test split; HELM\ncreates one with seed 121.\n\n## Who publishes it\n\nPekka Malo, Ankur Sinha, Pyry Takala, Pekka Korhonen, and Jyrki Wallenius\nat Aalto University School of Business. The journal version is JASIST 65\n(2014); the preprint is arXiv:1307.5336 (July 2013). Stanford CRFM wraps\nit for HELM. The Hub lastModified on the card fetched here is 2025-12-15.\n\n## Lineage\n\nAn early public finance-sentiment phrase set, still used because labelled\nfinance text was scarce in 2014. It is not a predecessor of FinQA or\nFinanceBench. No subset pages exist in this repository for the four\nagreement files.\n\n## Saturation and contamination\n\nThe sentences and labels have been public for more than a decade. HELM's\ntest split is a random 30% of the same public file. Treat high scores on\nmodels trained after 2014 as possibly contaminated. No current top HELM\ncell was read.\n\n## How to run it\n\nHELM: `financial_phrasebank` with `agreement` in {50, 66, 75, 100}. The\nrun name is `financial_phrasebank:agreement=50` at default. The scenario\nreads `Sentences_AllAgree.txt` or `Sentences_{agreement}Agree.txt` from the\nv1.0 zip pinned to Hub commit 598b6aad\u2026. Quote the agreement level. This\nid is not in the HELM Finance leaderboard entry list opened here (that\nlist has fin_qa, financebench, and banking77).\n\n## Reading the numbers\n\nA strong weighted F1 means the model named the investor-view label on\nHELM's shuffle of one agreement file. All-agree is smaller and cleaner;\n50% agree is larger and noisier. Weighted F1 is not accuracy and not a\nfinance-reasoning score. Compare only runs that share agreement, split\nseed, and metric. Read next to other finance tasks if the question is\nbroader domain skill.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "financial_phrasebank",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "FinancialPhrasebank",
   "Financial Phrase Bank"
  ],
  "category": "domain",
  "contamination": {
   "note": "Public since the 2014 journal paper (arXiv 2013), mirrored on Hugging Face and Kaggle, and small enough to appear in web-scale crawls. HELM's 30% \"test\" slice is a shuffle of the same public sentences, not a held out hidden set.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC-BY-NC-SA-3.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 4846,
   "size_note": "Hugging Face takala/financial_phrasebank sentences_50agree has 4,846 train rows; 66% agree 4,217; 75% agree 3,453; all-agree 2,264. The paper, Hub summary, and HELM scenario docstring all say the collection has 4,840 sentences. The 4,846 vs 4,840 gap is unresolved (see Dataset and licence). HELM default agreement is 50, so this page's size follows the Hub 50% config. Sixteen annotators; 5\u20138 labels per sentence.\n",
   "splits": "Hub configs have no official test split (all rows under train). HELM shuffles and uses 70% train / 30% test (random_seed 121) on one agreement file.",
   "url": "https://huggingface.co/datasets/takala/financial_phrasebank"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-043 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-043"
  },
  "harness": {
   "bigbench": "",
   "helm": "financial_phrasebank",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "HELM run spec function financial_phrasebank in enterprise_run_specs.py; run name financial_phrasebank:agreement={50,66,75,100}, default agreement=50. Scenario loads takala/financial_phrasebank zip at commit 598b6aad98f7c8d67be161b12a4b5f2497e07edd.\n"
  },
  "id": "financial_phrasebank",
  "last_updated": "2025-12",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "Financial PhraseBank asks whether an English sentence from financial news or a company release is positive, negative, or neutral from an investor's view of the stock, using only the sentence. Sentiment that is not about financial impact is labelled neutral. It is single-turn text classification, not numerical QA. Not [financebench](financebench.md), not [fin_qa](fin_qa.md), and not [financeiq](financeiq.md).\n",
  "metric": {
   "baseline_note": "HELM scenario metadata sets main_metric to classification_weighted_f1 on the test split. The enterprise run spec also attaches exact-match metrics and a ClassificationMetric over labels positive, neutral, negative (weighted F1, precision, recall). Three classes make uniform chance about 33%, but class balance is not uniform and HELM does not publish that figure as a baseline. No human-rater accuracy is recorded here.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1.0,
   "name": "classification_weighted_f1",
   "random_baseline": null,
   "unit": ""
  },
  "name": "Financial PhraseBank",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "1307.5336",
   "title": "Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts",
   "url": "https://arxiv.org/abs/1307.5336",
   "year": 2014
  },
  "publisher": {
   "authors": [
    "Pekka Malo",
    "Ankur Sinha",
    "Pyry Takala",
    "Pekka Korhonen",
    "Jyrki Wallenius"
   ],
   "org": "Aalto University School of Business",
   "url": "https://huggingface.co/datasets/takala/financial_phrasebank"
  },
  "released": "2014",
  "repo_url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/financial_phrasebank_scenario.py",
  "saturation": {
   "as_of": "",
   "note": "The set is widely used as a finance-sentiment training and test corpus. No current HELM Enterprise cell for financial_phrasebank was opened here, so no top_score is recorded.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM FinancialPhrasebankScenario (4840-sentence docstring, 70/30 split, weighted F1)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/financial_phrasebank_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM enterprise run spec financial_phrasebank:agreement= (default 50)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/run_specs/enterprise_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hub card (CC-BY-NC-SA-3.0, four agreement counts including 4846)",
    "url": "https://huggingface.co/datasets/takala/financial_phrasebank/raw/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hub API dataset_info split counts",
    "url": "https://huggingface.co/api/datasets/takala/financial_phrasebank"
   },
   {
    "accessed": "2026-09-08",
    "title": "Malo et al. arXiv abstract (submitted 2013-07-19; journal 2014)",
    "url": "https://arxiv.org/abs/1307.5336"
   }
  ],
  "status": "unknown",
  "subcategory": "three-class investor-view sentiment on English financial news sentences",
  "summary": "Aalto's three-class sentiment set of English financial-news sentences; HELM generates a label and reports weighted F1 on a 70/30 split.",
  "tags": [
   "finance",
   "sentiment",
   "classification",
   "helm"
  ],
  "task_format": "HELM generates a label string after instructions that list positive, neutral, and negative. The scenario shuffles the chosen agreement file with seed 121 and cuts a 70/30 train/test split. Original work has no canonical train/test split; Hugging Face exposes four agreement configs as a single train split each.\n"
 }
}