Hitachi's synthetic deduction set: given invented facts and a hypothesis, choose proved, disproved, or unknown without using world knowledge.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | synthetic multi-step formal-logic deduction (answer classification) |
| Page status | unknown |
| Metric | exact_match |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 5000 |
| Dataset licence | CC-BY-4.0 |
| Publisher | Hitachi, Ltd. (hitachi-nlp) |
FLD (Formal Logic Deduction) asks a model to (dis)prove a hypothesis from a set of facts that are logically structured but semantically invented, so memorised world knowledge should not help. The ICML 2023 paper scores both a proof trace and a three-way answer (proved / disproved / unknown). EleutherAI lm-eval ships the simplified "answer accuracy" setting only: generate the world_assump_label (Hub gold is uppercase PROVED, DISPROVED, or UNKNOWN). English natural-language facts, plus parallel logical-formula prompts. FLD★ (star / FLD.4) uses deeper trees (up to depth 8 vs up to 3) and is harder. Not [logical_deduction](logical_deduction.md) and not [formal_fallacies_syllogisms_negation](formal_fallacies_syllogisms_negation.md).
lm-eval default prompt: "Based on the provided facts ($context$), either prove or disprove the hypothesis or state that it is unknown." plus prompt_serial. Formula tasks substitute context_formula and hypothesis_formula. Target is world_assump_label. Metric is exact_match after stripping whitespace and taking the first line.
No model card in ModelSpec reports this benchmark yet.