Feedbackqa Eval

feedbackqa-eval

Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deployment — Li et al. (2022) (arXiv:2204.03025, 2022)

What this evaluates

This benchmark evaluates retrieval-based question answering systems and their ability to incorporate post-deployment user feedback. It probes a model's capacity to retrieve relevant answer passages, generate human-like explanations for answer quality, and rerank candidate answers using interactive feedback signals.

Datasets

• FEEDBACKQA — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Top-1 accuracy of answer selection after reranking. Candidates are scored by summing the RQA model's retrieval probability and the reranker's probability of assigning an 'Excellent' rating. The candidate with the highest combined score is selected as the final answer.

Input / output format

Input: A question Q and a set of candidate answer passages A (or a single candidate A for the reranker).

Output: For the reranker: a rating label Y ∈ {Excellent, Acceptable, Could be improved, Bad} and a natural language explanation E. For evaluation: a ranked list of candidates and a binary correctness flag for the top-1 selection.

Scoring recipe

# For each candidate A in A_set:
rqa_score = P_theta(A | Q, A_set)  # from Eq 1
reranker_score = P_phi(y="excellent" | Q, A)  # from Eq 2/3
combined_score = rqa_score + reranker_score
# Rank candidates by combined_score descending
predicted_answer = top_1_candidate
accuracy = 1.0 if predicted_answer == gold_answer else 0.0

Common pitfalls

• Using the normalized expectation of the rating score (0-3) instead of just the probability of 'Excellent' for reranking, which the authors found to be less effective.
• Failing to account for multiple annotator ratings per answer by using KL-divergence during training, which can lead to suboptimal reranker calibration.
• Treating the reranker as a joint model with the RQA retriever instead of decoupling them, which reduces reusability across different QA systems.

Evidence (verbatim from paper)

We simply sum up the scores from the RQA model and the reranker model. In practice, we found that using the reranker probability of excellent worked better than normalizing the expectation of the rating score (from score 0 for label bad to 3 for excellent). So, we score the candidate answers as follows: S(A | A, Q) = P_theta(A = A^+ | A, Q) + P_phi(y = excellent | A, Q)

Citation

@misc{li2022feedbackqa,
  title={Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deployment},
  author={Li et al. (2022)},
  year={2022},
  note={arXiv:2204.03025}
}

• arXiv: 2204.03025