qhjqhj00 label-accuracy-eval
Reframing Human-AI Collaboration for Generating Free-Text Explanations — Wiegreffe et al. (2021) (arXiv:2112.08674, 2021)
What this evaluates
Evaluates GPT-3's ability to predict ground-truth labels for given instances, and analyzes whether explanation quality correlates with prediction correctness across different datasets.
Datasets
- CommonsenseQA — total ?; splits: test (-1)
- SNLI — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of instances where the model's predicted label matches the ground-truth label.
Input / output format
Input: Question/context instance formatted as a prompt, explicitly excluding the 'why?' token and any gold explanations.
Output: A single predicted label corresponding to the instance.
Scoring recipe
correct = 0
for pred, gold in zip(predictions, gold_labels):
if pred == gold:
correct += 1
accuracy = (correct / len(gold_labels)) * 100
Common pitfalls
- Low accuracy on SNLI (46%) reflects known difficulties with sentence-comparison tasks, not necessarily poor explanation generation.
- High label accuracy does not guarantee that generated explanations are faithful or 'right for the right reasons'.
Evidence (verbatim from paper)
GPT-3 achieves 50.8% accuracy on CommonsenseQA compared to a 20% random baseline, and 46% accuracy on SNLI compared to a 33.33% random baseline.
Citation
@misc{wiegreffe2021reframing,
title={Reframing Human-AI Collaboration for Generating Free-Text Explanations},
author={Wiegreffe et al. (2021)},
year={2021},
note={arXiv:2112.08674}
}
- arXiv: 2112.08674