Zero Shot Commonsense Eval

zero-shot-commonsense-eval

Prompt Engineering and Calibration for Zero-Shot Commonsense Reasoning — Ma (2023) (arXiv:2304.06962, 2023)

What this evaluates

Evaluates zero-shot commonsense reasoning capabilities of language models using multiple-choice questions. It specifically probes how prompt engineering and probability calibration strategies affect accuracy across different model sizes and architectures.

Datasets

• CommonsenseQA — total ?; splits: dev (-1)
• COPA — total ?; splits: dev (-1)
• OpenBookQA — total ?; splits: dev (-1)
• PIQA — total ?; splits: dev (-1)
• Social IQA — total ?; splits: dev (-1)

Metrics

• accuracy (primary) — range: percent
  • Number of correctly predicted options divided by total number of instances, expressed as a percentage.

Input / output format

Input: Multiple-choice question with a set of candidate options.

Output: The model selects the option with the highest conditional probability score.

Scoring recipe

correct = 0
for instance in dataset:
    probs = [model.logit(option) for option in instance.options]
    pred = instance.options[argmax(probs)]
    if pred == instance.gold:
        correct += 1
accuracy = (correct / len(dataset)) * 100

Common pitfalls

• Combining prompt engineering and calibration often degrades performance rather than improving it due to negative interactions.
• Calibration benefits vary significantly across model architectures (e.g., helps GPT-2 but inconsistently helps T5).
• Prompt engineering only yields significant gains on larger, instruction-tuned models like Flan-T5-XL.

Evidence (verbatim from paper)

We evaluate prompt engineering and calibration on five multiple choice commonsense benchmarks: (1) CommonsenseQA (CSQA) (Talmor et al., 2019); (2) COPA (Gordon et al., 2012); (3) OpenBookQA (OBQA) (Mihaylov et al., 2018); (4)PIQA (Bisk et al., 2019); (5)Social IQA (SIQA) (Sap et al., 2019); We present their statistics in Appendix B. For all benchmarks, we only use their development sets. ... The evaluation metric is accuracy.

Citation

@misc{ma2023promptengineering,
  title={Prompt Engineering and Calibration for Zero-Shot Commonsense Reasoning},
  author={Ma (2023)},
  year={2023},
  note={arXiv:2304.06962}
}

• arXiv: 2304.06962