Trec Cast 2019 Eval

trec-cast-2019-eval

TREC CAsT 2019: The Conversational Assistance Track Overview — Dalton et al. (2020) (TREC 2019, 2020)

What this evaluates

Evaluates a system's ability to perform multi-turn conversational information retrieval by selecting relevant passages for user utterances while leveraging prior dialogue history, including handling coreference, omissions, and topic shifts.

Datasets

• TREC CAsT 2019 — total ?; splits: test (-1); repo https://github.com/microsoft/MSMARCO-Conversational-Search

Metrics

• NDCG@3 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 3. Computes the discounted gain of graded relevance scores at the top 3 positions, normalized by the ideal DCG@3 for the given relevance judgments.
• MAP — range: [0, 1]
  • Mean Average Precision. Averages the precision at each relevant document across all queries.
• MRR — range: [0, 1]
  • Mean Reciprocal Rank. Averages the reciprocal of the rank of the first relevant document across all queries.

Input / output format

Input: Multi-turn dialogue history (previous user utterances and system responses) plus the current user utterance (query).

Output: Ranked list of candidate text passages retrieved for the current utterance.

Scoring recipe

def compute_ndcg_at_3(predictions, gold):
    dcg = 0.0
    for i, doc_id in enumerate(predictions[:3]):
        rel = gold.get(doc_id, 0)
        dcg += (2**rel - 1) / math.log2(i + 2)
    ideal_rels = sorted(gold.values(), reverse=True)[:3]
    idcg = sum((2**r - 1) / math.log2(i + 2) for i, r in enumerate(ideal_rels))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Automatic runs use raw, unresolved queries with coreference/omissions, while manual runs use rewritten resolved queries; results differ significantly (~26% median gap).
• Pool incompleteness: only top 2 runs per group are pooled for assessment, leaving ~0.6-1.67 unjudged documents in top 10 per turn.
• Washington Post (WaPo) passages make up ~10% of pool candidates; filtering them affects some systems differently.

Evidence (verbatim from paper)

We use three standard TREC evaluation measures, Mean-average Precision (MAP), and Normalized Discounted Cumulative Gain (NDCG), and Mean Reciprocal Rank (MRR). In particular, we use NDCG@3 as the primary measure because we focus on graded relevance of results at the top ranks.

Citation

@misc{dalton2020trecast,
  title={TREC CAsT 2019: The Conversational Assistance Track Overview},
  author={Dalton et al. (2020)},
  year={2020},
  note={TREC 2019}
}

• arXiv: 2003.13624