QA Retrieval Eval

qa-retrieval-eval

MrRank: Improving Question Answering Retrieval System through Multi-Result Ranking Model — Khamnuansin et al. (2024) (arXiv:2406.05733, 2024)

What this evaluates

Evaluates the ability of retrieval models to rank relevant sentences or documents highest for a given question. It probes lexical and semantic matching capabilities in question answering contexts, testing both single-model retrieval and multi-model fusion strategies.

Datasets

• ReQA SQuAD — total 99025; splits: train (87599), dev (11426)
• ReQA NQ — total 75869; splits: train (74097), dev (1772)
• MTEB QA Subset — total ?; splits: dev (-1), test (-1)
• iapp-wiki-qa-squad — total ?; splits: test (-1)

Metrics

• MRR (primary) — range: [0, 1]
  • Mean Reciprocal Rank. For each query, compute 1/rank where rank is the position of the first relevant document/sentence. Average across all queries.
• R@1 — range: [0, 1]
  • Recall at 1. Proportion of queries where the first retrieved document/sentence is relevant.
• R@5 — range: [0, 1]
  • Recall at 5. Proportion of queries where at least one relevant document/sentence appears in the top 5 results.
• R@10 — range: [0, 1]
  • Recall at 10. Proportion of queries where at least one relevant document/sentence appears in the top 10 results.
• EM — range: [0, 1]
  • Exact Match. Binary score indicating whether the predicted answer string exactly matches the ground truth answer string.
• ROUGE-L — range: [0, 1]
  • Longest Common Subsequence based F1 score measuring overlap between predicted and reference answers.
• METEOR — range: [0, 1]
  • Metric for Evaluation of Translation with Explicit ORdering, measuring alignment between predicted and reference text using synonyms and stemming.
• BERT-F1 — range: [0, 1]
  • F1 score computed using BERT token embeddings to measure semantic overlap between predicted and reference answers.

Input / output format

Input: A question (query) and a pool of candidate sentences or documents to retrieve from.

Output: A ranked list of candidate sentences/documents ordered by relevance to the question.

Scoring recipe

def compute_mrr(relevant_ranks):
    return sum(1.0 / r for r in relevant_ranks) / len(relevant_ranks)

def compute_recall_at_k(relevant_ranks, k):
    return sum(1 if r <= k else 0 for r in relevant_ranks) / len(relevant_ranks)

# For QA metrics (EM, ROUGE-L, METEOR, BERT-F1):
# Run retrieved top-k documents through a reader model (e.g., BERT) to extract answers.
# Compare extracted answer against ground truth using respective metric functions.

Common pitfalls

• Uses the development set as the evaluation set for ReQA SQuAD and ReQA NQ, not the held-out test set.
• Focuses on sentence-level retrieval rather than document-level, which changes the candidate pool size and granularity.
• Ensemble baselines (RRF, routing) require careful score normalization before fusion; naive averaging can skew rankings.

Evidence (verbatim from paper)

We use the Mean Reciprocal Rank (MRR) and Recall at k (R@k) as evaluation metrics. The experiments were replicated five times, and the mean value is reported. The development set was used as the evaluation set for both datasets. We focused on the sentence-level retrieval task, following the method outlined in the original ReQA paper.

Citation

@misc{khamnuansin2024mrrank,
  title={MrRank: Improving Question Answering Retrieval System through Multi-Result Ranking Model},
  author={Khamnuansin et al. (2024)},
  year={2024},
  note={arXiv:2406.05733}
}

• arXiv: 2406.05733