Squad V1.1 Eval

squad-v1.1-eval

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — Devlin et al. (2018) (arXiv:1810.04805, 2018)

What this evaluates

Measures extractive question answering capability by requiring the model to identify a text span in a passage that answers a given question. It tests precise token-level span prediction and contextual understanding.

Datasets

• SQuAD v1.1 — total 100000; splits: train (-1), dev (-1), test (-1); HF squad

Metrics

• exact-match (EM) (primary) — range: [0, 1]
  • Percentage of questions where the predicted answer string exactly matches any of the ground truth answer strings.
• F1 — range: [0, 1]
  • Token-level F1 score between the predicted answer and the ground truth answer, averaged over all questions.

Input / output format

Input: Question and passage packed into a single sequence with distinct segment embeddings (A for question, B for passage).

Output: A start position and an end position for the answer span within the passage.

Scoring recipe

def compute_metrics(preds, golds):
    em = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    f1s = [token_f1(p, g) for p, g in zip(preds, golds)]
    return em, sum(f1s)/len(f1s)

Common pitfalls

• The model predicts start/end logits over all tokens; the best span is chosen by maximizing start+end scores, not by independently picking max start and max end.
• Ensembling multiple models with different seeds/checkpoints significantly boosts performance compared to single models.

Evidence (verbatim from paper)

The Stanford Question Answering Dataset (SQuAD v1.1) is a collection of 100k crowdsourced question/answer pairs... The score of a candidate span from position i to position j is defined as S·T_i + E·T_j, and the maximum scoring span where j ≥ i is used as a prediction. ... Table 2: SQuAD 1.1 results. ... EM | F1

Citation

@misc{devlin2018bert,
  title={BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding},
  author={Devlin et al. (2018)},
  year={2018},
  note={arXiv:1810.04805}
}

• arXiv: 1810.04805