Convdr Eval

convdr-eval

Few-Shot Conversational Dense Retrieval — Shi Yu et al. (2021) (arXiv:2105.04166, 2021)

What this evaluates

Evaluates conversational dense retrieval models on their ability to rank relevant documents given multi-turn conversational queries. It probes context capture, few-shot learning effectiveness, and robustness to noisy conversation history compared to query rewriting baselines.

Datasets

• TREC CAsT — total ?; splits: test (-1); repo https://github.com/thunlp/ConvDR
• OR-QuAC — total ?; splits: test (-1)

Metrics

• NDCG@3 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 3. Computes the weighted sum of relevance scores for the top 3 retrieved documents, normalized by the ideal DCG@3.
• MRR@5 (primary) — range: [0, 1]
  • Mean Reciprocal Rank at rank 5. Takes the reciprocal of the rank of the first relevant document in the top 5 results, averaged over queries.
• Win/Tie/Loss (%) — range: percent
  • Pairwise comparison metric reporting the percentage of queries where the model wins, ties, or loses against a baseline based on ranking accuracy.

Input / output format

Input: Multi-turn conversational query history concatenated with the current user query, paired with a candidate document.

Output: A ranked list of documents based on dot-product similarity scores, optionally reranked by a cross-encoder (e.g., BERT).

Scoring recipe

def compute_ndcg_at_k(relevance_scores, k=3):
    dcg = sum(r / (2**i - 1) for i, r in enumerate(relevance_scores[:k], 1))
    ideal = sorted(relevance_scores, reverse=True)[:k]
    idcg = sum(r / (2**i - 1) for i, r in enumerate(ideal, 1))
    return dcg / idcg if idcg > 0 else 0.0

def compute_mrr_at_k(relevance_scores, k=5):
    for i, r in enumerate(relevance_scores[:k], 1):
        if r > 0: return 1.0 / i
    return 0.0

Common pitfalls

• Confusing first-stage dense retrieval scores with reranking scores; the paper evaluates both separately and notes performance drops in reranking.
• Overlooking the 'hole rate' on CAsT, which indicates missing relevant documents and suggests reported metrics may underestimate true performance.
• Assuming synthetic OR-QuAC labels reflect real user feedback; the paper warns that artifacts in synthetic data over-simplify retrieval.

Evidence (verbatim from paper)

Table 4: Results of different training paradigms. Candidates from ConvDR (KD) and ConvDR (Multi-Task) are reranked with BERT on CAsT-19 and OR-QuAC. Win/Tie/Loss (%) are compared with ANCE-Manual or ANCE-Manual→BERT-Manual. | Method | CAsT 2019 | | OR-QuAC | | | NDCG@3 | W/T/L | MRR@5 | W/T/L |

Citation

@misc{shi2021fewshotconversational,
  title={Few-Shot Conversational Dense Retrieval},
  author={Shi Yu et al. (2021)},
  year={2021},
  note={arXiv:2105.04166}
}

• arXiv: 2105.04166