Cosmos QA Eval

cosmos-qa-eval

Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning — Huang et al. (2019) (arXiv:1909.00277, 2019)

What this evaluates

This benchmark evaluates a model's ability to perform contextual commonsense reasoning in machine reading comprehension. It probes whether systems can make non-literal, implicit inferences about causes, effects, and counterfactuals based on personal narratives, rather than relying on explicit textual evidence or simple semantic matching.

Datasets

• Cosmos QA — total 35600; splits: dev (-1), test (-1)

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly predicted answers out of the total number of questions. Calculated as (number of correct predictions / total questions) × 100.

Input / output format

Input: A context paragraph, a question about the paragraph, and a set of multiple-choice candidate answers.

Output: A single selected candidate answer from the provided options.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Models often overfit to lexical/semantic overlap between the paragraph and distractor answers, failing to perform actual commonsense inference.
• The dataset contains >83% questions where the correct answer is not explicitly stated in the text, so retrieval or exact-match strategies will fail.

Evidence (verbatim from paper)

Table 3: Comparison of varying approaches (Accuracy %). ... Most of the reading comprehension approaches apply attention to capture the correlation between paragraph, question and each candidate answer and tend to select the answer which is the most semantically closed to the paragraph.

Citation

@misc{huang2019cosmosqa,
  title={Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning},
  author={Huang et al. (2019)},
  year={2019},
  note={arXiv:1909.00277}
}

• arXiv: 1909.00277