Convsearch R1 Eval

convsearch-r1-eval

ConvSearch-R1: Enhancing Query Reformulation for Conversational Search with Reasoning via Reinforcement Learning — Zhu et al. (2025) (arXiv:2505.15776, 2025)

What this evaluates

Evaluates conversational query reformulation (CQR) by measuring how effectively a model rewrites multi-turn queries into standalone search queries that retrieve relevant passages. It probes the model's ability to optimize rewrites using only retrieval signals, without human annotations or LLM distillation.

Datasets

• TopiOCQA — total ?; splits: test (-1)
• QReCC — total ?; splits: test (-1)

Metrics

• MRR@3 (primary) — range: [0, 1]
  • Mean Reciprocal Rank at cutoff 3. For each query, compute 1/rank of the first relevant document in the top 3 results, then average across all queries.
• NDCG@3 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at cutoff 3. Measures the quality of the ranked list by discounting relevance scores logarithmically by position and normalizing by the ideal DCG.
• Recall@K — range: [0, 1]
  • Fraction of relevant documents retrieved within the top K results. Computed as |relevant_retrieved| / |total_relevant|.

Input / output format

Input: Conversational dialogue history (previous turns) and the current user query.

Output: A single rewritten query string optimized for retrieval.

Scoring recipe

def compute_metrics(ranked_docs, relevant_docs, k=3):
    # ranked_docs: list of doc IDs in retrieval order
    # relevant_docs: set of ground truth relevant doc IDs
    
    # MRR@3
    rr = 0.0
    for i, doc in enumerate(ranked_docs[:k]):
        if doc in relevant_docs:
            rr = 1.0 / (i + 1)
            break
    mrr = rr
    
    # NDCG@3
    dcg = sum(rel / math.log2(i + 2) for i, rel in enumerate([1 if d in relevant_docs else 0 for d in ranked_docs[:k]]))
    idcg = sum(1 / math.log2(i + 2) for i in range(min(len(relevant_docs), k)))
    ndcg = dcg / idcg if idcg > 0 else 0.0
    
    # Recall@K
    recall = len(set(ranked_docs[:k]) & relevant_docs) / len(relevant_docs) if relevant_docs else 0.0
    
    return mrr, ndcg, recall

Common pitfalls

• The paper evaluates using both BM25 and ANCE retrievers without retraining them, so results are highly sensitive to the pre-existing retrieval index quality and passage granularity.
• Recall@K requires a fixed K value (commonly 20 or 100 in conversational search benchmarks), which is not explicitly specified in the setup section and must be chosen consistently.
• Rewritten queries must be evaluated as standalone queries; failing to strip conversational context or pronouns before retrieval will artificially lower scores.

Evidence (verbatim from paper)

For metrics, we adopt MRR@3, NDCG@3, and Recall@K (referred to as R@K in this paper) for evaluation.

Citation

@misc{zhu2025convsearchr1,
  title={ConvSearch-R1: Enhancing Query Reformulation for Conversational Search with Reasoning via Reinforcement Learning},
  author={Zhu et al. (2025)},
  year={2025},
  note={arXiv:2505.15776}
}

• arXiv: 2505.15776