Dlmia Eval

dlmia-eval

Understanding the User: An Intent-Based Ranking Dataset — Anand et al. (2024) (arXiv:2408.17103, 2024)

What this evaluates

Evaluates ranking models' ability to align machine-generated relevance with fine-grained user intents, particularly for ambiguous or multi-intent queries. It also measures the diversity of search results when multiple user intents are fused into a single ranking.

Datasets

• DL-MIA — total ?; splits: test (-1)

Metrics

• α-nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10, weighted by intent relevance (alpha). Measures both relevance and diversity by assigning weights to different user intents per query and normalizing against an ideal ranking.

Input / output format

Input: Query string and a candidate set of passages/documents.

Output: A ranked list of passages/documents.

Scoring recipe

def compute_alpha_ndcg_at_10(relevance_scores, intent_weights, k=10):
    # relevance_scores: list of doc relevance scores per intent
    # intent_weights: list of intent importance weights
    dcg = sum(w * (2**r - 1) / log2(i + 2) for i, (r, w) in enumerate(zip(relevance_scores, intent_weights)) if i < k)
    ideal_scores = sorted(relevance_scores, reverse=True)
    idcg = sum(w * (2**r - 1) / log2(i + 2) for i, (r, w) in enumerate(zip(ideal_scores, intent_weights)) if i < k)
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Evaluating on original queries versus explicit user intents yields different performance; the protocol requires testing both settings separately.
• When fusing multiple intent rankings, Reciprocal Rank Fusion must use k=60 to generate the unified ranking for the original query.
• Multi-intent queries require handling overlapping or merged intents during evaluation, which can skew diversity metrics if not properly weighted.

Evidence (verbatim from paper)

We additionally demonstrate the diversity ranking performance of various models using the α-nDCG@10 metric. To achieve this in the second setting (where user intents are treated as queries), we employ reciprocal rank fusion [6] with k = 60. This technique is applied to the intent-based rankings to generate a unified ranking for the original query.

Citation

@misc{anand2024dlmia,
  title={Understanding the User: An Intent-Based Ranking Dataset},
  author={Anand et al. (2024)},
  year={2024},
  note={arXiv:2408.17103}
}

• arXiv: 2408.17103