Commonsenseqa Eval

commonsenseqa-eval

CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge — Talmor et al. (2018) (arXiv:1811.00937, 2018)

What this evaluates

This benchmark evaluates a model's ability to answer multiple-choice questions that require real-world commonsense knowledge. It specifically probes whether models can distinguish a correct answer from semantically plausible but factually incorrect distractors based on spatial, causal, or physical reasoning.

Datasets

• CommonsenseQA — total ?; splits: train (-1), dev (-1), test (-1)

Metrics

• accuracy (primary) — range: percent
  • The proportion of examples for which the model's prediction matches the gold label. Calculated as the number of correct predictions divided by the total number of examples.

Input / output format

Input: A question prompt accompanied by multiple-choice options, including one correct answer and several semantically plausible distractors derived from ConceptNet.

Output: A single selected option from the provided multiple-choice list.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
accuracy = (correct / len(gold_labels)) * 100

Common pitfalls

• Using the 'question concept split' instead of the primary 'random split' artificially inflates performance because models can memorize concept-answer mappings that appear in both train and test sets.
• Confusing the main benchmark with the 'SANITY' mode, which replaces hard distractors with random ConceptNet ones to establish a much easier baseline.
• Assuming retrieval-augmented approaches (e.g., using web snippets) significantly improve performance, as the paper demonstrates they yield only marginal gains.

Evidence (verbatim from paper)

We evaluate all models on the test set using accuracy (proportion of examples for which prediction is correct), and tune hyper-parameters for all trained models on the development set.

Citation

@misc{talmor2018commonsenseqa,
  title={CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge},
  author={Talmor et al. (2018)},
  year={2018},
  note={arXiv:1811.00937}
}

• arXiv: 1811.00937