# Zero Shot Commonsense Eval

> Evaluates zero-shot commonsense reasoning capabilities of language models using multiple-choice questions. It specifically probes how prompt engineering and probability calibration strategies affect accuracy across different model sizes and architectures. Use when the user wants to benchmark on CommonsenseQA, COPA, OpenBookQA, PIQA, Social IQA, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/zero-shot-commonsense-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/zero-shot-commonsense-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/zero-shot-commonsense-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/zero-shot-commonsense-eval

---


# zero-shot-commonsense-eval

> Prompt Engineering and Calibration for Zero-Shot Commonsense Reasoning — Ma (2023) (arXiv:2304.06962, 2023)

## What this evaluates

Evaluates zero-shot commonsense reasoning capabilities of language models using multiple-choice questions. It specifically probes how prompt engineering and probability calibration strategies affect accuracy across different model sizes and architectures.

## Datasets

- **CommonsenseQA** — total ?; splits: dev (-1)
- **COPA** — total ?; splits: dev (-1)
- **OpenBookQA** — total ?; splits: dev (-1)
- **PIQA** — total ?; splits: dev (-1)
- **Social IQA** — total ?; splits: dev (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Number of correctly predicted options divided by total number of instances, expressed as a percentage.

## Input / output format

**Input**: Multiple-choice question with a set of candidate options.

**Output**: The model selects the option with the highest conditional probability score.

## Scoring recipe

```python
correct = 0
for instance in dataset:
    probs = [model.logit(option) for option in instance.options]
    pred = instance.options[argmax(probs)]
    if pred == instance.gold:
        correct += 1
accuracy = (correct / len(dataset)) * 100
```

## Common pitfalls

- Combining prompt engineering and calibration often degrades performance rather than improving it due to negative interactions.
- Calibration benefits vary significantly across model architectures (e.g., helps GPT-2 but inconsistently helps T5).
- Prompt engineering only yields significant gains on larger, instruction-tuned models like Flan-T5-XL.

## Evidence (verbatim from paper)

> We evaluate prompt engineering and calibration on five multiple choice commonsense benchmarks: (1) CommonsenseQA (CSQA) (Talmor et al., 2019); (2) COPA (Gordon et al., 2012); (3) OpenBookQA (OBQA) (Mihaylov et al., 2018); (4)PIQA (Bisk et al., 2019); (5)Social IQA (SIQA) (Sap et al., 2019); We present their statistics in Appendix B. For all benchmarks, we only use their development sets. ... The evaluation metric is accuracy.

## Citation

```bibtex
@misc{ma2023promptengineering,
  title={Prompt Engineering and Calibration for Zero-Shot Commonsense Reasoning},
  author={Ma (2023)},
  year={2023},
  note={arXiv:2304.06962}
}
```

- arXiv: 2304.06962