Tanda As2 Eval

tanda-as2-eval

TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection — Garg et al. (2019) (arXiv:1911.04118, 2019)

What this evaluates

Evaluates a model's ability to rank candidate answer sentences for a given question. It specifically probes the stability and robustness of pre-trained transformer models when transferred to a target domain using a two-stage fine-tuning protocol.

Datasets

• WikiQA — total ?; splits: train (873), dev (126), test (243)
• TREC-QA — total ?; splits: train (1229), dev (65), test (68)

Metrics

• MAP (primary) — range: [0, 1]
  • Mean Average Precision. Computes the average of precision values at the ranks where relevant items occur, averaged over all queries.
• MRR — range: [0, 1]
  • Mean Reciprocal Recall. The reciprocal of the rank of the first relevant item in the ranked list of candidates.

Input / output format

Input: A question paired with a list of candidate answer sentences.

Output: A ranked list of the candidate answer sentences, ordered by predicted relevance to the question.

Scoring recipe

def compute_map_mrr(ranked_preds, gold_labels):
    # gold_labels: binary relevance (1 if correct, 0 otherwise)
    # ranked_preds: list of candidates sorted by predicted score descending
    first_rel_idx = next((i for i, rel in enumerate(gold_labels) if rel == 1), -1)
    mrr = 1.0 / (first_rel_idx + 1) if first_rel_idx != -1 else 0.0
    num_rel = sum(gold_labels)
    if num_rel == 0:
        ap = 0.0
    else:
        precisions = []
        rel_count = 0
        for i, rel in enumerate(gold_labels):
            if rel == 1:
                rel_count += 1
                precisions.append(rel_count / (i + 1))
        ap = sum(precisions) / num_rel
    return ap, mrr

Common pitfalls

• Using the 'raw' WikiQA split instead of the standard 'no all-' for training and 'clean' for testing.
• Evaluating on the transfer datasets (ASNQ/QNLI) instead of the target evaluation datasets (WikiQA/TREC-QA).
• Not using the entire set of candidate sentences per question during evaluation, as explicitly required by the protocol.

Evidence (verbatim from paper)

Metrics We measure system accuracy with Mean Average Precision (MAP) and Mean Reciprocal Recall (MRR) evaluated on the test set, using the entire set of candidates for each questions (this varies according to the different datasets).

Citation

@misc{garg2019tanda,
  title={TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection},
  author={Garg et al. (2019)},
  year={2019},
  note={arXiv:1911.04118}
}

• arXiv: 1911.04118