Conv Search Rewriting Eval

conv-search-rewriting-eval

Ask Optimal Questions: Aligning Large Language Models with Retriever's Preference in Conversation — Yoon et al. (2024) (arXiv:2402.11827, 2024)

What this evaluates

This evaluation probes a model's ability to rewrite conversational search queries to maximize retrieval effectiveness. It measures how well reformulated questions help both sparse and dense retrievers locate relevant passages across different dialogue contexts, including initial turns and topic shifts.

Datasets

• QReCC — total 14000; splits: test (-1)
• TopiOCQA — total 3900; splits: test (-1)

Metrics

• MRR (primary) — range: [0, 1]
  • Mean Reciprocal Rank: the average of 1/rank across all queries, where rank is the position of the first relevant passage in the retrieved list.
• R@10 — range: [0, 1]
  • Recall@10: the fraction of queries where at least one relevant passage appears in the top-10 retrieved results.
• R@100 — range: [0, 1]
  • Recall@100: the fraction of queries where at least one relevant passage appears in the top-100 retrieved results.
• NDCG@3 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at 3: evaluates retrieval results by considering both relevance and rank of the top-3 results, normalized by the ideal DCG.

Input / output format

Input: Dialogue history (context) and the current user turn/question.

Output: A single rewritten query string optimized for retrieval.

Scoring recipe

def compute_metrics(predictions, gold_passages, retriever, k_values=[10, 100]):
    mrr_scores = []
    recall_counts = {k: 0 for k in k_values}
    for pred, gold in zip(predictions, gold_passages):
        ranked_docs = retriever.search(pred)
        rank = next((i+1 for i, doc in enumerate(ranked_docs) if doc in gold), len(ranked_docs)+1)
        mrr_scores.append(1.0 / rank)
        for k in k_values:
            if any(doc in gold for doc in ranked_docs[:k]):
                recall_counts[k] += 1
    mrr = sum(mrr_scores) / len(mrr_scores)
    recalls = {k: v / len(predictions) for k, v in recall_counts.items()}
    return mrr, recalls

Common pitfalls

• Failing to segment evaluation by dialogue turn type (initial, topic-concentrated, topic-shifted), which significantly impacts TopiOCQA results and masks model robustness.
• Assuming retriever preference alignment generalizes perfectly; models trained on BM25 preferences may underperform on dense retrievers like ANCE without specific tuning.
• Over-specifying queries to improve MRR or R@10 can inadvertently hurt Recall@100 due to longer, more detailed rewrites that miss loosely matched passages.

Evidence (verbatim from paper)

Mean Reciprocal Rank (MRR) is the average of the ranks measuring how effectively the retriever can locate gold passages. Normalized Discounted Cumulative Gain (NDCG@3) evaluates retrieval results by considering both relevance and rank of top-3 results. Recall@$k$ verifies whether the retriever succeeds in locating gold passages within top-$k$ results.

Citation

@misc{yoon2024askoptimalquestions,
  title={Ask Optimal Questions: Aligning Large Language Models with Retriever's Preference in Conversation},
  author={Yoon et al. (2024)},
  year={2024},
  note={arXiv:2402.11827}
}

• arXiv: 2402.11827