qhjqhj00 cqa-eval
Explain Yourself! Leveraging Language Models for Commonsense Reasoning — Rajani et al. (2019) (arXiv:1906.02361, 2019)
What this evaluates
Evaluates a model's ability to answer multiple-choice commonsense reasoning questions. It probes whether providing natural language explanations (human or model-generated) alongside questions improves reasoning performance compared to a baseline without explanations.
Datasets
- CQA — total ?; splits: dev-random-split (-1), test (-1)
Metrics
Accuracy (%)(primary) — range: percent- Percentage of correctly predicted answer choices out of the total number of examples. Calculated as (number of correct predictions / total examples) × 100.
Input / output format
Input: Multiple-choice question with answer choices. In experimental variants, this is concatenated with a natural language explanation (human-generated from CoS-E or auto-generated via CAGE).
Output: A single predicted answer choice from the provided options.
Scoring recipe
def compute_accuracy(predictions, gold_labels):
correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
return (correct / len(gold_labels)) * 100
Common pitfalls
- Using human explanations at inference time creates an oracle setting that leaks ground-truth information, inflating performance unrealistically.
- Auto-generated explanations may inadvertently contain words from the answer choices, artificially boosting accuracy through lexical overlap rather than genuine reasoning.
- Transferring explanations from CQA to other commonsense datasets (SWAG, Story Cloze) yields negligible or negative gains, indicating explanation transfer is non-trivial.
Evidence (verbatim from paper)
Table 2: Results on CQA dev-random-split with CoS-E used during training. Table 2 shows results that compare a BERT baseline that uses only the CQA inputs and the same architecture but trained using inputs that contain explanations from CoS-E during training. The BERT baseline model reaches 64% accuracy and adding open-ended human explanations (CoS-E-open-ended) alongside the questions during training results in a 2% boost in accuracy. By generating explanations as described in Section 4.1, we can give the commonsense question answering model access to an explanation that is not conditioned on the ground truth. These explanations (CAGE-reasoning) can be provided during both training and validation and increases the accuracy to 72%.
Citation
@misc{rajani2019explainyourself,
title={Explain Yourself! Leveraging Language Models for Commonsense Reasoning},
author={Rajani et al. (2019)},
year={2019},
note={arXiv:1906.02361}
}
- arXiv: 1906.02361