Five-way multiple-choice flight-booking task that tests compositional and conditional reasoning over user constraints; 4,849 public test items.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | compositional and conditional reasoning via five-way flight-option matching |
| Page status | active |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 4849 |
| Dataset licence | CC-BY-4.0 |
| Publisher | Ohio State University NLP Group |
GroundCocoa asks a model to pick which of five scraped flight options satisfies a long English user request. Requests are built from 2–6 flight slots (airline, times, price, layovers, carbon, and others) combined as a product-of-sums formula, then paraphrased with GPT-4 Turbo and checked by hand. The skill is if-then and multi-constraint grounding, not dialogue state tracking and not a travel agent with tools. Atypical queries include odd wants such as more layovers or above-average carbon.
Five-way multiple choice. lm-evaluation-harness task groundcocoa formats the query and options A–E and scores log-likelihood of "The answer is Option {A–E}". output_type multiple_choice. metric acc. Paper also reports zero-shot, chain-of-thought (full and partial), and least-to-most prompting. L2M numbers in the paper use a 200-item subset.
No model card in ModelSpec reports this benchmark yet.