Econlogicqa Eval

econlogicqa-eval

EconLogicQA: A Question-Answering Benchmark for Evaluating Large Language Models in Economic Sequential Reasoning — Quan et al. (2024) (arXiv:2405.07938, 2024)

What this evaluates

This benchmark evaluates large language models' ability to perform economic sequential reasoning by logically ordering interconnected business and supply chain events. It probes multi-event causality and temporal reasoning beyond simple chronological sorting, requiring models to understand complex economic narratives.

Datasets

• EconLogicQA — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: percent
  • Exact match accuracy between the model's predicted event sequence and the ground-truth sequence. Calculated as the number of correctly sequenced instances divided by the total number of instances.

Input / output format

Input: A prompt containing 1 or 5 few-shot examples of economic event sequencing tasks, followed by the target question requiring the model to order a set of events.

Output: A sequentially ordered list of events. The model's raw text output is parsed using a regular expression to extract the final answer.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return (correct / len(golds)) * 100

Common pitfalls

• 0-shot evaluation yields unsatisfactory results; the protocol explicitly requires 1-shot or 5-shot settings.
• Increasing the number of shots beyond 5 or applying Chain-of-Thought prompting in 1-shot settings can degrade performance or provide no benefit.
• Permuting the order of few-shot examples in the prompt significantly impacts models like GPT-4-Turbo, reducing accuracy in 1-shot scenarios.

Evidence (verbatim from paper)

Accuracy is the primary metric used, offering a direct measure of each model’s understanding of the concepts within the EconLogicQA dataset. ... Then, we extract the answer from the response generated by LLMs using regular expression. Finally, we verify the answer through exact matching and use accuracy as the evaluation metric.

Citation

@misc{quan2024econlogicqa,
  title={EconLogicQA: A Question-Answering Benchmark for Evaluating Large Language Models in Economic Sequential Reasoning},
  author={Quan et al. (2024)},
  year={2024},
  note={arXiv:2405.07938}
}

• arXiv: 2405.07938