{
 "body": "\n## What it measures\n\nFLD checks multi-step deduction when the facts are fake. Each item gives a\ncontext of invented statements and a hypothesis. The model must decide\nwhether the hypothesis is proved, disproved, or unknown from those facts\nalone. Because the predicates have no real-world meaning, looking up \"who\nis whose grandson\" cannot help. The paper also asks for a proof sequence;\nlm-eval does not.\n\nTwo English Hub configs exist. `default` is the basic FLD.v2 dump (FLD.3 in\nthe paper, trees up to depth 3). `star` is FLD\u2605 / FLD.4 (trees up to depth\n8). Formula tasks keep the same labels but write facts as predicate calculus\n(`&`, `v`, `\u00ac`, `->`). JFLD is a later Japanese benchmark and is not this id.\n\n## How it is scored\n\nlm-eval uses exact_match on `world_assump_label` after\n`remove_whitespace` and `take_first`. Hub gold is uppercase PROVED,\nDISPROVED, or UNKNOWN; the prompt asks for prove/disprove/unknown, so case\ncan fail the match. That is the paper's \"answer accuracy\" setting, not proof\naccuracy. Chance is 33.3% with three labels. Table 4 of the ICML PDF reports\n10-shot numbers on a protocol that still asks for proofs: GPT-4 answer\naccuracy 52.4 (FLD) and 49.4 (FLD\u2605), proof accuracy 12.8 and 3.2. Those are\nnot drop-in substitutes for a zero-shot `fld_default` run. Fine-tuned T5 in\nthe same table (91.6 / 72.2 answer accuracy) is a trained prover.\n\n## Dataset and licence\n\n`hitachi-nlp/FLD.v2` holds 30,000 / 5,000 / 5,000 rows for each of default\nand star. This page's size is the 5,000-row test split that lm-eval scores.\nThe FLD-corpus repository licences the data under CC-BY-4.0. The FLD code\nrepository is Apache-2.0; that licence does not replace the corpus licence.\nThe Hub card itself has no licence tag.\n\n## Who publishes it\n\nHitachi NLP: Terufumi Morishita, Gaku Morio, Atsuki Yamaguchi, and\nYasuhiro Sogawa. The evaluation paper is ICML 2023 (PMLR v202,\narXiv:2308.07336). A NeurIPS 2024 follow-up introduces FLDx2 as a training\ncorpus; it is not the lm-eval task.\n\n## Lineage\n\nSynthetic deduction relative to RuleTaker-style work, but this page is the\nHitachi FLD.v2 / lm-eval suite. Related in this repository:\n[logical_deduction](logical_deduction.md) (object ordering) and\n[formal_fallacies_syllogisms_negation](formal_fallacies_syllogisms_negation.md)\n(valid vs invalid syllogisms). Neither is a subset of FLD. JFLD and FLDx2\ndo not yet have pages.\n\n## Saturation and contamination\n\nGPT-4 at about half on answer accuracy in 2023 is not a ceiling, and no\ncurrent lm-eval top score was read. Proof accuracy in the paper is much\nlower (12.8 / 3.2 for GPT-4). The authors note GPT-4 rarely answers\n\"unknown\". Test strings are public; world knowledge is not the intended\nleak path.\n\n## How to run it\n\n`lm_eval --tasks fld_default` (and `fld_star`,\n`fld_logical_formula_default`, `fld_logical_formula_star`). Quote the YAML\n`task:` name. The README's `fld_logical_formula_fld_star` string does not\nmatch the YAML. Do not mix answer-only exact_match with proof accuracy.\n\n## Reading the numbers\n\nA strong `fld_default` exact_match means the model emitted the Hub gold\nstring (PROVED, DISPROVED, or UNKNOWN) on 5,000 public test rows. It does\nnot mean the proof was valid.\n`fld_star` is a harder depth setting, not a second shuffle of the same\nitems. Formula tasks drop English. Compare T5-finetuned numbers only when\nthe reporter trained on FLD. Read beside other knowledge-free logic tasks,\nnot beside MMLU formal logic.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "fld",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Formal Logic Deduction",
   "FLD.v2",
   "FLD-star",
   "FLD\u2605"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "Facts are synthetic and counterfactual, so ordinary pretraining knowledge should not solve items. The 5,000-row test splits have been public on Hugging Face since August 2023, so the strings themselves can still leak into later training data.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC-BY-4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 5000,
   "size_note": "Hugging Face hitachi-nlp/FLD.v2, configs default and star, each has train 30,000 / validation 5,000 / test 5,000 (card and dataset-viewer agree). lm-eval names training_split, validation_split and test_split; the scored split is test (5,000 per task). Formula tasks reuse the same rows with formula fields. Corpus README: these are version-2.0 dumps of the ICML English corpora (FLD.3 default, FLD.4 star), detailed in Appendix H.\n",
   "splits": "per config: train 30,000 / validation 5,000 / test 5,000; lm-eval scores test",
   "url": "https://huggingface.co/datasets/hitachi-nlp/FLD.v2"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-043 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-043"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "fld_default",
   "opencompass": "",
   "other": "Runnable YAML task names: fld_default (dataset_name default), fld_star (star), fld_logical_formula_default, fld_logical_formula_star. The README also says group fld and names fld_logical_formula_fld_star; no group: key exists in the YAML files, and the star formula task field is fld_logical_formula_star. Official Hitachi eval scripts exist but are a different path.\n"
  },
  "id": "fld",
  "last_updated": "2023-12",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "FLD (Formal Logic Deduction) asks a model to (dis)prove a hypothesis from a set of facts that are logically structured but semantically invented, so memorised world knowledge should not help. The ICML 2023 paper scores both a proof trace and a three-way answer (proved / disproved / unknown). EleutherAI lm-eval ships the simplified \"answer accuracy\" setting only: generate the world_assump_label (Hub gold is uppercase PROVED, DISPROVED, or UNKNOWN). English natural-language facts, plus parallel logical-formula prompts. FLD\u2605 (star / FLD.4) uses deeper trees (up to depth 8 vs up to 3) and is harder. Not [logical_deduction](logical_deduction.md) and not [formal_fallacies_syllogisms_negation](formal_fallacies_syllogisms_negation.md).\n",
  "metric": {
   "baseline_note": "Three labels, so uniform chance is 33.3%, matching Table 4 \"random guess\" for answer accuracy. The paper also reports proof accuracy (random 0.0). GPT-4 10-shot answer accuracy in Table 4: 52.4 on FLD and 49.4 on FLD\u2605; proof accuracy 12.8 and 3.2. Fine-tuned T5 is much higher (91.6 / 72.2 answer accuracy) and is not a zero-shot LLM baseline. No human-rater figure is given.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "exact_match",
   "random_baseline": 33.3,
   "unit": "%"
  },
  "name": "FLD (Formal Logic Deduction)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2308.07336",
   "title": "Learning Deductive Reasoning from Synthetic Corpus based on Formal Logic",
   "url": "https://arxiv.org/abs/2308.07336",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Terufumi Morishita",
    "Gaku Morio",
    "Atsuki Yamaguchi",
    "Yasuhiro Sogawa"
   ],
   "org": "Hitachi, Ltd. (hitachi-nlp)",
   "url": "https://github.com/hitachi-nlp/FLD"
  },
  "released": "2023-07",
  "repo_url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/fld",
  "saturation": {
   "as_of": "",
   "note": "The paper's own claim is that even GPT-4 solves only about half the answer-accuracy items under 10-shot (52.4 / 49.4). That is a 2023 figure on the paper's protocol (proof-plus-answer), not a current lm-eval leaderboard cell, so top_score is left empty.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "lm-eval fld README (answer-accuracy setting, four task names)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/fld/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "fld_default.yaml (hitachi-nlp/FLD.v2 default, exact_match)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/fld/fld_default.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "fld_star.yaml (dataset_name star)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/fld/fld_star.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "fld_logical_formula_default.yaml (formula prompt, exact_match)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/fld/fld_logical_formula_default.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "fld_logical_formula_star.yaml (task: fld_logical_formula_star)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/fld/fld_logical_formula_star.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face FLD.v2 card (30k/5k/5k default and star)",
    "url": "https://huggingface.co/datasets/hitachi-nlp/FLD.v2"
   },
   {
    "accessed": "2026-09-08",
    "title": "FLD-corpus README (PROVED/DISPROVED/UNKNOWN, v2.0, Appendix H)",
    "url": "https://raw.githubusercontent.com/hitachi-nlp/FLD-corpus/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "FLD-corpus Creative Commons Attribution 4.0 International",
    "url": "https://raw.githubusercontent.com/hitachi-nlp/FLD-corpus/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "hitachi-nlp/FLD (Apache-2.0 code; NeurIPS_2024 default branch)",
    "url": "https://github.com/hitachi-nlp/FLD"
   },
   {
    "accessed": "2026-09-08",
    "title": "ICML 2023 paper HTML (Table 4 GPT-4 10-shot numbers)",
    "url": "https://ar5iv.labs.arxiv.org/html/2308.07336"
   },
   {
    "accessed": "2026-09-08",
    "title": "FLD paper abstract (arXiv:2308.07336)",
    "url": "https://arxiv.org/abs/2308.07336"
   },
   {
    "accessed": "2026-09-08",
    "title": "PMLR v202 Morishita et al. ICML 2023",
    "url": "https://proceedings.mlr.press/v202/morishita23a.html"
   }
  ],
  "status": "unknown",
  "subcategory": "synthetic multi-step formal-logic deduction (answer classification)",
  "summary": "Hitachi's synthetic deduction set: given invented facts and a hypothesis, choose proved, disproved, or unknown without using world knowledge.",
  "tags": [
   "formal-logic",
   "deductive-reasoning",
   "synthetic",
   "exact-match",
   "three-way-classification"
  ],
  "task_format": "lm-eval default prompt: \"Based on the provided facts ($context$), either prove or disprove the hypothesis or state that it is unknown.\" plus prompt_serial. Formula tasks substitute context_formula and hypothesis_formula. Target is world_assump_label. Metric is exact_match after stripping whitespace and taking the first line.\n"
 }
}