Mctaco Eval

mctaco-eval

"Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understanding — Ben Zhou et al. (2019) (arXiv:1909.03065, 2019)

What this evaluates

Evaluates a model's ability to reason about temporal commonsense, including event duration, ordering, typical time, frequency, and stationarity. It tests whether systems can correctly classify candidate answers as 'likely' or 'unlikely' given a context sentence and a question.

Datasets

• MCTACO — total ?; splits: dev (-1), test (-1)

Metrics

• exact match (EM) — range: percent
  • Fraction of questions where the model's set of predicted 'likely' answers exactly matches the gold set of 'likely' answers. Formula: EM = (1/|D|) * sum_{q in D} prod_{a in q} f(a;q).
• F1 (primary) — range: percent
  • Macro-averaged F1 score across all questions. Question-level F1 is the harmonic mean of precision and recall for predicting 'likely' answers. Formula: F1 = (1/|D|) * sum_{q in D} F1(q).

Input / output format

Input: Concatenation of a context sentence and a temporal commonsense question, paired with a single candidate answer.

Output: Binary classification label: 'likely' or 'unlikely' for each candidate answer.

Scoring recipe

em_scores = []
f1_scores = []
for q in dataset:
    gold_likely = {a for a in q.candidates if a.is_gold}
    pred_likely = {a for a in q.candidates if a.prediction == 'likely'}
    em_scores.append(1.0 if gold_likely == pred_likely else 0.0)
    if len(pred_likely) == 0:
        f1_scores.append(0.0)
    else:
        prec = len(gold_likely & pred_likely) / len(pred_likely)
        rec = len(gold_likely & pred_likely) / len(gold_likely)
        f1_scores.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0)
return {'EM': sum(em_scores)/len(em_scores), 'F1': sum(f1_scores)/len(f1_scores)}

Common pitfalls

• EM requires perfect prediction of all 'likely' answers for a question; a single missed or extra 'likely' prediction yields 0 for that question.
• The task is binary classification per candidate ('likely'/'unlikely'), not selecting a single correct answer from a multiple-choice list.
• Human baseline is not 100% due to subjective commonsense; gold labels are crowdsourced, so strict exact-match evaluation can be overly punitive for reasonable alternative temporal interpretations.

Evidence (verbatim from paper)

Two question-level metrics are adopted in this work: exact match (EM) and F1 . ... EM measures how many questions a system is able to correctly label all candidate answers, while F1 is more relaxed and measures the average overlap between one's predictions and the ground truth.

Citation

@misc{zhou2019going,
  title={"Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understanding},
  author={Ben Zhou et al. (2019)},
  year={2019},
  note={arXiv:1909.03065}
}

• arXiv: 1909.03065