Selqa Eval

selqa-eval

SelQA: A New Benchmark for Selection-based Question Answering — Jurczyk et al. (2016) (arXiv:1606.08513, 2016)

What this evaluates

Evaluates a model's ability to retrieve relevant answer sentences from a document given a question (selection), and to determine whether a document section contains an answer at all (triggering). It probes open-domain QA robustness against paraphrasing, varying question types, and section lengths.

Datasets

• SelQA — total 7904; splits: train (5529), dev (785), test (1590)

Metrics

• MAP (primary) — range: [0, 1]
  • Mean Average Precision: the average of precision values computed at each rank where a relevant sentence is retrieved.
• MRR — range: [0, 1]
  • Mean Reciprocal Rank: the average of the reciprocal of the rank of the first relevant sentence retrieved for each question.
• F1-score — range: [0, 1]
  • Question-level F1-score: harmonic mean of precision and recall calculated at the question level for the binary answer triggering task.

Input / output format

Input: A natural language question paired with a set of candidate sentences (for selection) or sections (for triggering).

Output: A ranked list of candidate sentences (selection) or a binary decision per section indicating whether it contains the answer (triggering).

Scoring recipe

def compute_mrr(preds, golds):
    ranks = []
    for pred, gold in zip(preds, golds):
        for i, p in enumerate(pred):
            if p == gold:
                ranks.append(1.0 / (i + 1))
                break
    return sum(ranks) / len(ranks)

def compute_f1(preds, golds):
    tp = sum(1 for p, g in zip(preds, golds) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(preds, golds) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(preds, golds) if p == 0 and g == 1)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• Models may exploit word overlap between questions and answers, leading to inflated development set performance compared to the evaluation set.
• Confusing the two tasks: Answer Sentence Selection requires ranking sentences, while Answer Triggering is a binary classification at the section level.
• Reporting development set scores as final results instead of using the held-out evaluation set.

Evidence (verbatim from paper)

Our systems are evaluated for the answer sentence selection and answer triggering tasks on both WikiQA and our corpus. Two metrics are used, mean average precision (MAP) and mean reciprocal rank (MRR), for the evaluation of this task. ... Thus, the F1-score on the question level was proposed by [2] as the evaluation for this task, which we follow.

Citation

@misc{jurczyk2016selqa,
  title={SelQA: A New Benchmark for Selection-based Question Answering},
  author={Jurczyk et al. (2016)},
  year={2016},
  note={arXiv:1606.08513}
}

• arXiv: 1606.08513