Trec Cast21 Eval

trec-cast21-eval

From Baseline to Top Performer: A Reproducibility Study of Approaches at the TREC 2021 Conversational Assistance Track — Lajewska et al. (2023) (arXiv:2301.10493, 2023)

What this evaluates

Evaluates conversational search retrieval pipelines by reproducing baseline and top-performing systems from the TREC CAsT 2021 track. It probes the effectiveness of query rewriting, sparse/dense retrieval fusion, and re-ranking strategies in multi-turn conversational settings.

Datasets

• TREC CAsT 2021 — total ?; splits: test (-1)

Metrics

• NDCG@3 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 3. Measures the quality of the top-3 ranked documents, discounting relevance by position and normalizing by the ideal ranking.
• Recall@500 — range: [0, 1]
  • Fraction of relevant documents (binary relevance threshold ≥ 2) retrieved within the top 500 ranked passages.
• MAP — range: [0, 1]
  • Mean Average Precision across conversational turns, averaging precision at each relevant document's rank.
• MRR — range: [0, 1]
  • Mean Reciprocal Rank, averaging the inverse rank of the first relevant document per turn.
• NDCG@500 — range: [0, 1]
  • Normalized Discounted Cumulative Gain computed with a rank cutoff of 500.

Input / output format

Input: Multi-turn conversational queries with history (previous queries and canonical responses) for query rewriting; document passages from the TREC CAsT index for retrieval and re-ranking.

Output: Ranked list of document passages (initially top 1000 candidates, then final ranked list).

Scoring recipe

def compute_metrics(predictions, gold, cutoff=3, recall_cutoff=500, rel_thresh=2):
    import math
    rel_scores = [gold.get(d, 0) for d in predictions[:cutoff]]
    dcg = sum(s / math.log2(i + 2) for i, s in enumerate(rel_scores))
    ideal = sorted(rel_scores, reverse=True)
    idcg = sum(s / math.log2(i + 2) for i, s in enumerate(ideal))
    ndcg = dcg / idcg if idcg > 0 else 0.0
    rel_in_top = sum(1 for d in predictions[:recall_cutoff] if gold.get(d, 0) >= rel_thresh)
    total_rel = sum(1 for s in gold.values() if s >= rel_thresh)
    recall = rel_in_top / total_rel if total_rel > 0 else 0.0
    return {'NDCG@3': ndcg, 'Recall@500': recall}

Common pitfalls

• Missing documentation on query rewriting input formatting and model length handling, leading to trimmed inputs or removed raw queries.
• Undisclosed QA system and PRF algorithm details for the WaterlooClarke baseline, causing significant performance drops during reproduction.
• BM25 parameter sensitivity to document preprocessing and index construction, making reported parameters suboptimal without exact preprocessing details.
• Discrepancies between officially reported track results and direct runfile evaluation, questioning the reliability of published baselines.

Evidence (verbatim from paper)

Following the official setup, we consider measures with both binary and graded relevance. The main measure is NDCG@3; other measures are computed with a rank cutoff of 500. For binary measures, we apply a relevance threshold of 2.

Citation

@misc{lajewska2023reproducibility,
  title={From Baseline to Top Performer: A Reproducibility Study of Approaches at the TREC 2021 Conversational Assistance Track},
  author={Lajewska et al. (2023)},
  year={2023},
  note={arXiv:2301.10493}
}

• arXiv: 2301.10493