{
 "body": "\n## What it measures\n\nGroundCocoa is a five-option flight-booking quiz. The model sees a user wish-list and five flight records. Only one record fits every constraint. Wishes are not independent filters: slots are tied with AND/OR so that one attribute depends on another. That is the compositional and conditional load. Time, price, and carbon constraints also need ordinary arithmetic and comparison.\n\nThe authors scrape Google Flights among busy airports, sample 2\u20136 slots, build a minterm table, simplify it with SymPy, verbalise rules, paraphrase with GPT-4 Turbo, and verify by hand. The same wish can appear with different option sets. A slice of items uses atypical primitives (want more layovers, higher price, or extra carbon) to punish \u201cusual trip\u201d priors.\n\n## How it is scored\n\nThe paper reports accuracy of the chosen letter. Direct prompting, CoT with five explanations, CoT with two explanations, and least-to-most are separate columns. L2M is marked on a 200-item subset. lm-evaluation-harness implements `groundcocoa` as multiple-choice log-likelihood of `The answer is Option X`, which is not CoT. Chance is 20%. No human accuracy is published.\n\nTable 3 (arXiv v2) puts GPT-4 Turbo + CoT-partial at 66.92% overall. Llama 3.1 70B chat sits near 58\u201360%. Mixtral 8x7B Instruct is 44.48% direct. Small Llama-2 chat models sit near chance. CoT is mixed: it helps GPT-4 Turbo a little and can hurt Mixtral.\n\n## Dataset and licence\n\nHugging Face `harsh147/GroundCocoa` is CC-BY-4.0. Test has 4,849 rows (728 unique queries). Validation has 52 rows (6 unique queries). The GitHub repo has generation and `run_eval.py` but no LICENSE file; the card licence is the one recorded here. Answers are public.\n\n## Who publishes it\n\nHarsh Kohli, Sachin Kumar, and Huan Sun at Ohio State University. arXiv v1 is 5 April 2024; v2 is 13 February 2025. The paper is NAACL 2025 main. The project page, GitHub, and Hugging Face card are the official surfaces. There is no separate numeric leaderboard beyond the paper table.\n\n## Lineage\n\nThe paper contrasts GroundCocoa with ConditionalQA, RuleTaker, ProofWriter, LogicNLI, FOLIO, and schema-guided dialogue. None of those ids is this task. It is not [gsm8k](gsm8k.md) and not a tool-using travel agent. No family page exists in this repository.\n\n## Saturation and contamination\n\nBest reported accuracy is still in the mid-60s on five-way choice, so the set still separates models as of the 2025 paper. Public labels and LLM-written queries are a medium leakage path. Atypical items drop GPT-4 Turbo by several points in Table 3, which is a robustness check, not a contamination test.\n\n## How to run it\n\nlm-eval: `--tasks groundcocoa` on `harsh147/GroundCocoa` with streaming. The YAML uses validation and test; there is no train split. Paper numbers need the authors' `run_eval.py` and the matching CoT or L2M prompt. Do not compare lm-eval accuracy to L2M-on-200-items.\n\n## Reading the numbers\n\nA 60% score is well above chance and still far from solved. It means the model can align nested constraints to a schema more often than not, not that it can book a real ticket. CoT and L2M can move the number either way. Unique-query count is 728, so option-set repeats can inflate apparent diversity. Read regular versus atypical columns before calling a model robust.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "groundcocoa",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "GroundCocoa",
   "ground_cocoa",
   "harsh147/GroundCocoa"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "Test answers are public on Hugging Face (CC-BY-4.0). Items are synthetic POS expansions over scraped Google Flights pages, not copied exam questions. Queries were LLM-paraphrased. The paper isolates atypical primitives to reduce reliance on common booking text. No contamination study of the 4,849 items was opened here.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC-BY-4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 4849,
   "size_note": "Hugging Face harsh147/GroundCocoa and the project site: 4,849 test samples from 728 unique user requirements (same query, different option sets) plus 52 validation samples from 6 unique requirements. datasets-server confirms validation 52 / test 4,849. Paper Table 1 HTML on ar5iv is malformed; card counts are used. Queries were paraphrased with GPT-4 Turbo and then manually checked.\n",
   "splits": "validation (52) / test (4,849); lm-eval uses both, training_split null",
   "url": "https://huggingface.co/datasets/harsh147/GroundCocoa"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-046 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-046"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "groundcocoa",
   "opencompass": "",
   "other": "Authors' run_eval.py for GPT-4 Turbo CoT variants; dataset_path harsh147/GroundCocoa."
  },
  "id": "groundcocoa",
  "last_updated": "2025-02",
  "leaderboard_url": "https://osu-nlp-group.github.io/GroundCocoa/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "GroundCocoa asks a model to pick which of five scraped flight options satisfies a long English user request. Requests are built from 2\u20136 flight slots (airline, times, price, layovers, carbon, and others) combined as a product-of-sums formula, then paraphrased with GPT-4 Turbo and checked by hand. The skill is if-then and multi-constraint grounding, not dialogue state tracking and not a travel agent with tools. Atypical queries include odd wants such as more layovers or above-average carbon.\n",
  "metric": {
   "baseline_note": "Uniform chance on five options is 20%. No human accuracy is reported in the paper or dataset card. Table 3 best figure: GPT-4 Turbo with CoT-partial 66.92% overall (67.77% regular / 65.62% atypical). The abstract says even GPT-4 Turbo did not exceed 67% despite advanced prompting; that rounds the 66.92% overall / 67.77% regular pair.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 20.0,
   "unit": "%"
  },
  "name": "GroundCocoa",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2404.04237",
   "title": "GroundCocoa: A Benchmark for Evaluating Compositional & Conditional Reasoning in Language Models",
   "url": "https://aclanthology.org/2025.naacl-long.420/",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Harsh Kohli",
    "Sachin Kumar",
    "Huan Sun"
   ],
   "org": "Ohio State University NLP Group",
   "url": "https://osu-nlp-group.github.io/GroundCocoa/"
  },
  "released": "2024-04",
  "repo_url": "https://github.com/OSU-NLP-Group/GroundCocoa",
  "saturation": {
   "as_of": "2025-02",
   "note": "Paper Table 3 (arXiv v2, 13 Feb 2025 / NAACL 2025): GPT-4 Turbo + CoT-partial 66.92% on the full test mix. Llama 3.1 70B chat is 58\u201360% in the same table. No later public leaderboard cell was read.\n",
   "status": "open",
   "top_score": 66.92
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "GroundCocoa arXiv:2404.04237 (v1 2024-04-05, v2 2025-02-13; comments: NAACL 2025 Main)",
    "url": "https://arxiv.org/abs/2404.04237"
   },
   {
    "accessed": "2026-09-08",
    "title": "Kohli, Kumar, and Sun, NAACL 2025 (anthology 2025.naacl-long.420, pp. 8280\u20138295)",
    "url": "https://aclanthology.org/2025.naacl-long.420/"
   },
   {
    "accessed": "2026-09-08",
    "title": "GroundCocoa full text (pipeline, Table 3 accuracies)",
    "url": "https://ar5iv.labs.arxiv.org/html/2404.04237"
   },
   {
    "accessed": "2026-09-08",
    "title": "harsh147/GroundCocoa dataset card (CC-BY-4.0, 4849/52, 728 unique queries)",
    "url": "https://huggingface.co/datasets/harsh147/GroundCocoa"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face dataset API (license cc-by-4.0, lastModified 2025-02-28)",
    "url": "https://huggingface.co/api/datasets/harsh147/GroundCocoa"
   },
   {
    "accessed": "2026-09-08",
    "title": "GroundCocoa project page",
    "url": "https://osu-nlp-group.github.io/GroundCocoa/"
   },
   {
    "accessed": "2026-09-08",
    "title": "OSU-NLP-Group/GroundCocoa repository",
    "url": "https://github.com/OSU-NLP-Group/GroundCocoa"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval groundcocoa README",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/groundcocoa/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval groundcocoa.yaml (task: groundcocoa, metric acc)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/groundcocoa/groundcocoa.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval process_docs prompt template",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/groundcocoa/utils.py"
   }
  ],
  "status": "active",
  "subcategory": "compositional and conditional reasoning via five-way flight-option matching",
  "summary": "Five-way multiple-choice flight-booking task that tests compositional and conditional reasoning over user constraints; 4,849 public test items.",
  "tags": [
   "compositional-reasoning",
   "conditional-reasoning",
   "multiple-choice",
   "flight-booking",
   "lm-eval"
  ],
  "task_format": "Five-way multiple choice. lm-evaluation-harness task groundcocoa formats the query and options A\u2013E and scores log-likelihood of \"The answer is Option {A\u2013E}\". output_type multiple_choice. metric acc. Paper also reports zero-shot, chain-of-thought (full and partial), and least-to-most prompting. L2M numbers in the paper use a 200-item subset.\n"
 }
}