Groundcocoa Eval

groundcocoa-eval

GroundCocoa: A Benchmark for Evaluating Compositional & Conditional Reasoning in Language Models — Kohli et al. (2024) (arXiv:2404.04237, 2024)

What this evaluates

Evaluates compositional and conditional reasoning in LLMs by requiring them to match complex, logically constrained user preferences to specific flight booking options. It probes the model's ability to handle interdependent requirements and atypical constraints without external reasoning engines.

Datasets

• GroundCocoa — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly matched flight options to user requirements. Calculated as (correct predictions / total samples) * 100.

Input / output format

Input: A user requirement encoded as logically constrained, interdependent preferences, followed by 5 flight options.

Output: Selection of the flight option(s) that satisfy the user requirement.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
accuracy = (correct / len(gold_labels)) * 100

Common pitfalls

• Context length limits can degrade CoT performance; CoT-partial (2 options) often outperforms CoT-full (5 options) due to confusion between requirements and options.
• Least-to-Most prompting scales turns linearly with complexity, leading to high inference costs and potential error accumulation.
• Models show strong bias toward typical user needs, causing significant accuracy drops when atypical/unconventional constraints are introduced.

Evidence (verbatim from paper)

To measure performance on GroundCocoa, we test several models of different sizes including both open-source and closed-source LLMs - LLAMA 2-chat (Touvron et al., [2023]) / LLAMA 3-Instruct (Dubey et al., [2024]), Mixtral 8x7B - Instruct (Jiang et al., [2024]) / Mistral 7B Instruct (Jiang et al., [2023]), Gemini Pro (Team et al., [2023]), and GPT-4 Turbo. Results from our experiments are shown in Table[3]. We have 3 different evaluation setups for the our models - direct prompting, chain-of-thought (CoT) (Wei et al., [2022]) prompting, and least-to-most (L2M) prompting (Zhou et al. ([2023])). Table 3: Accuracy (%) on GroundCocoa.

Citation

@misc{kohli2024groundcocoa,
  title={GroundCocoa: A Benchmark for Evaluating Compositional & Conditional Reasoning in Language Models},
  author={Kohli et al. (2024)},
  year={2024},
  note={arXiv:2404.04237}
}

• arXiv: 2404.04237