Squad2.0 Eval

squad2.0-eval

Know What You Don't Know: Unanswerable Questions for SQuAD — Rajpurkar et al. (2018) (arXiv:1806.03822, 2018)

What this evaluates

Probes a model's ability to perform extractive reading comprehension while correctly identifying when a question cannot be answered from the provided context. It forces models to distinguish between answerable and unanswerable questions, testing knowledge gap detection and resistance to semantically relevant distractors.

Datasets

• SQuAD 2.0 — total ?; splits: dev (-1), test (-1)

Metrics

• exact match — range: [0, 1]
  • 1 if the predicted answer exactly matches the gold answer, 0 otherwise.
• F1 (primary) — range: [0, 1]
  • Harmonic mean of token-level precision and recall between the predicted answer span and the gold answer span.

Input / output format

Input: A context paragraph and a question.

Output: A predicted answer span extracted from the context, or a decision to abstain (predict unanswerable) based on a learned probability threshold.

Scoring recipe

def compute_metrics(preds, golds):
    em, f1 = 0, 0
    for p, g in zip(preds, golds):
        if p is None:  # abstained
            continue
        em += 1 if normalize(p) == normalize(g) else 0
        f1 += token_f1(normalize(p), normalize(g))
    return em / len(golds), f1 / len(golds)

Common pitfalls

• Models frequently fail to abstain on unanswerable questions, instead predicting the crowdworker-provided plausible distractor.
• The abstention threshold is tuned on the development set to maximize F1, meaning test scores are sensitive to this calibration step.
• F1 is computed at the token level, not character level, which can yield different scores than character-level exact match baselines.

Evidence (verbatim from paper)

Following Rajpurkar et al. (2016), we report average exact match and F1 scores. $^{3}$ The best model, DocQA + ELMo, achieves only 66.3 F1 on the test set, 23.2 points lower than the human accuracy of 89.5 F1.

Citation

@misc{rajpurkar2018squad2,
  title={Know What You Don't Know: Unanswerable Questions for SQuAD},
  author={Rajpurkar et al. (2018)},
  year={2018},
  note={arXiv:1806.03822}
}

• arXiv: 1806.03822