Squad Fquad Eval

squad-fquad-eval

EfficientQA : a RoBERTa Based Phrase-Indexed Question-Answering System — Chaybouti et al. (2021) (arXiv:2101.02157, 2021)

What this evaluates

Evaluates a model's ability to extract and rank answer candidates from a given context for phrase-indexed question answering. It probes both the quality of candidate retrieval and the accuracy of final answer selection against gold spans.

Datasets

• SQuAD v1.1 — total 100000; splits: train (87599), dev (10570); HF squad
• FQuAD — total 26399; splits: train (20731), dev (5668); HF fquad

Metrics

• exact-match (primary) — range: [0, 1]
  • 1 if the predicted answer string exactly matches any of the gold answer strings, else 0.
• f1-score — range: [0, 1]
  • Token-level F1 score between the predicted answer and the best-matching gold answer string.

Input / output format

Input: A natural language question and a context passage.

Output: A ranked list of extracted answer candidates (phrases) from the context; the top-ranked candidate is taken as the final prediction.

Scoring recipe

def compute_metrics(predictions, golds):
    em, f1 = 0.0, 0.0
    for pred, gold_list in zip(predictions, golds):
        em += 1.0 if pred in gold_list else 0.0
        pred_tok = pred.split()
        f1 += max(token_f1(pred_tok, g.split()) for g in gold_list)
    return em / len(predictions), f1 / len(predictions)

Common pitfalls

• The paper evaluates exact-match and F1 over all extracted candidates rather than just the top-1 prediction, which measures retrieval recall rather than final answer accuracy.
• Direct comparison with vanilla fine-tuned QA baselines ignores the fundamental difference between span-extraction and phrase-indexed retrieval paradigms.

Evidence (verbatim from paper)

Our model was trained on the train set ($87599$ pairs) and evaluated on the development set ($10570$ pairs). The architectures are evaluated with exact-match and f1-score over all selected candidates.

Citation

@misc{chaybouti2021efficientqa,
  title={EfficientQA : a RoBERTa Based Phrase-Indexed Question-Answering System},
  author={Chaybouti et al. (2021)},
  year={2021},
  note={arXiv:2101.02157}
}

• arXiv: 2101.02157