Squad V2.0 Eval

squad-v2.0-eval

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — Devlin et al. (2018) (arXiv:1810.04805, 2018)

What this evaluates

Extends extractive QA by allowing questions that have no answer in the passage, testing the model's ability to abstain or predict a null span. It evaluates robustness against unanswerable questions.

Datasets

• SQuAD v2.0 — total ?; splits: train (-1), dev (-1), test (-1); HF squad_v2

Metrics

• F1 (primary) — range: [0, 1]
  • Token-level F1 score, extended to handle null answers. Predicted null is compared against gold null or text spans appropriately.

Input / output format

Input: Question and passage packed into a single sequence. The [CLS] token represents the null answer span.

Output: A start position, an end position, and a threshold comparison against a null score to decide between a text span and a null answer.

Scoring recipe

def compute_f1_v2(preds, golds):
    best_span = max(preds, key=lambda x: x.start_score + x.end_score)
    null_score = best_span.cls_score
    threshold = tune_threshold_on_dev(preds, golds)
    if best_span.score > null_score + threshold:
        pred_answer = best_span.text
    else:
        pred_answer = ""
    return squad_v2_f1_score(pred_answer, golds)

Common pitfalls

• A threshold τ must be tuned on the dev set to maximize F1 when deciding between a null answer and a text span.
• The null answer is represented by the [CLS] token, and its score is compared against the best non-null span score.

Evidence (verbatim from paper)

The SQuAD 2.0 task extends the SQuAD 1.1 problem definition by allowing for the possibility that no short answer exists in the provided paragraph... We predict a non-null answer when ŝ_i,j > s_null + τ, where the threshold τ is selected on the dev set to maximize F1. ... We observe a +5.1 F1 improvement over the previous best system.

Citation

@misc{devlin2018bert,
  title={BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding},
  author={Devlin et al. (2018)},
  year={2018},
  note={arXiv:1810.04805}
}

• arXiv: 1810.04805