Unconfounded Propensity Estimation Eval

unconfounded-propensity-estimation-eval

Unconfounded Propensity Estimation for Unbiased Ranking — Dan Luo et al. (2023) (arXiv:2305.09918, 2023)

What this evaluates

This protocol evaluates unbiased learning-to-rank models on their ability to correct position bias and propensity overestimation using implicit click feedback. It probes ranking quality under both dynamic online and static offline logging policies by comparing predicted rankings against ground truth relevance.

Datasets

• Yahoo! LETOR — total 29921; splits: train (19944), val (2994), test (6983)
• Istella-S — total 33000; splits: train (-1), val (-1), test (-1)

Metrics

• NDCG@K (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at cutoff K. It computes the discounted gain of predicted relevance scores relative to an ideal ranking, normalized by the ideal DCG. Standard convention uses log2(i+2) for position discounting.
• ERR@K — range: [0, 1]
  • Expected Reciprocal Rank at cutoff K. It estimates the probability that a user stops at a relevant document at or before position K, assuming a stopping probability proportional to document relevance.

Input / output format

Input: Per query, a ranked list of top N=10 documents with feature vectors (700-D for Yahoo!, 220-D for Istella-S) and ground truth 5-level relevance labels (0-4).

Output: Predicted ranking scores or propensity estimates for each document, used to generate a ranked ordering for evaluation.

Scoring recipe

def compute_ndcg_at_k(gold_relevance, pred_scores, k=10):
    ranked_pairs = sorted(zip(pred_scores, gold_relevance), key=lambda x: x[0], reverse=True)
    dcg = sum((2**rel - 1) / math.log2(i + 2) for i, (_, rel) in enumerate(ranked_pairs[:k]))
    ideal_rels = sorted(gold_relevance, reverse=True)
    idcg = sum((2**rel - 1) / math.log2(i + 2) for i, rel in enumerate(ideal_rels[:k]))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Training relies on synthetically simulated clicks generated via a Position Bias Model (η=1, ε=0.1), but evaluation must strictly use ground truth relevance labels, not simulated clicks.
• IPW-Random is theoretically an upper bound but often underperforms in practice due to high variance from result randomization; it should not be assumed to strictly dominate other methods.
• Statistical significance is determined via Fisher randomization test (p≤0.05), not standard parametric tests.

Evidence (verbatim from paper)

To evaluate all methods, we use the normalized Discounted Cumulative Gain (nDCG) and the Expected Reciprocal Rank (ERR). For both metrics, we report the results at ranks 1, 3, 5, and 10 to show the performance of models on different positions.

Citation

@misc{luo2023unconfounded,
  title={Unconfounded Propensity Estimation for Unbiased Ranking},
  author={Dan Luo et al. (2023)},
  year={2023},
  note={arXiv:2305.09918}
}

• arXiv: 2305.09918