Crown Eval

crown-eval

CROWN: Conversational Passage Ranking by Reasoning over Word Networks — Kaiser et al. (2019) (arXiv:1911.02850, 2019)

What this evaluates

Evaluates conversational passage ranking by measuring how effectively a model ranks relevant documents across multi-turn search queries, balancing term similarity with contextual coherence.

Datasets

• TREC CAsT 2019 — total ?; splits: train (-1), test (-1); repo https://github.com/magkai/CROWN

Metrics

• nDCG (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain: ranks predictions by relevance, applies logarithmic discounting based on position, and normalizes by the ideal DCG.
• ERR — range: [0, 1]
  • Expected Reciprocal Rank: models user browsing behavior by calculating the probability of stopping at each position based on relevance.
• AP — range: [0, 1]
  • Average Precision: computes the mean of precision values at ranks where relevant documents occur.

Input / output format

Input: Multi-turn conversational queries with associated candidate passages retrieved from a combined collection of MS MARCO, TREC CAR, and Washington Post documents.

Output: A ranked list of candidate passages (top 1000 retrieved by Indri, then re-ranked by the model's scoring function).

Scoring recipe

def compute_ndcg(relevance_scores, k=1000):
    dcg = sum(rel / log2(i + 2) for i, rel in enumerate(relevance_scores[:k]))
    ideal = sorted(relevance_scores, reverse=True)
    idcg = sum(rel / log2(i + 2) for i, rel in enumerate(ideal[:k]))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• The document collection is a custom combination of MS MARCO, TREC CAR, and Washington Post, not a single standard corpus.
• Hyperparameters like node threshold (α), edge threshold (β), and weights (h1, h2, h3) are model tuning settings, not evaluation criteria.
• Evaluation uses graded relevance scores rather than binary relevance, requiring metrics like nDCG/ERR instead of simple accuracy.

Evidence (verbatim from paper)

Since responses are assessed using graded relevance, we used nDCG (normalized discounted cumulative gain) and ERR (expected reciprocal rank) as metrics. Furthermore, AP (average precision) is reported on the evaluation data.

Citation

@misc{kaiser2019crown,
  title={CROWN: Conversational Passage Ranking by Reasoning over Word Networks},
  author={Kaiser et al. (2019)},
  year={2019},
  note={arXiv:1911.02850}
}

• arXiv: 1911.02850