Coat Ranking Eval

coat-ranking-eval

Towards Robust Offline Evaluation: A Causal and Information Theoretic Framework for Debiasing Ranking Systems — Khatami et al. (2025) (arXiv:2504.03997, 2025)

What this evaluates

Evaluates the ability of debiasing frameworks to transform biased (MNAR) recommendation data into unbiased (MAR) representations, measuring how well debiased rankings align with ground-truth user preferences. It probes whether reweighting or perturbation mechanisms successfully mitigate selection and staleness biases without degrading predictive performance.

Datasets

• Coat — total 11600; splits: train (6960), test (4640)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve. Measures the probability that a randomly chosen positive instance (rating ≥4) is ranked higher than a randomly chosen negative instance (rating <4).
• Precision — range: [0, 1]
  • Fraction of predicted positive clicks that are actually positive: TP / (TP + FP).
• Recall — range: [0, 1]
  • Fraction of actual positive clicks that are correctly predicted: TP / (TP + FN).
• F1 — range: [0, 1]
  • Harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall).
• Wasserstein Distance — range: other
  • Earth Mover's Distance between the predicted click distribution P(C|X^r) and the true distribution P(C|X^r, X^{nr}). Quantifies the distributional shift after debiasing.

Input / output format

Input: User and item feature vectors (X^r), exposure/bias features (X^{nr}), and propensity scores. Explicit 1-5 ratings are binarized as positive (≥4) or negative (<4) for evaluation.

Output: Predicted click probability or ranking score for each user-item pair.

Scoring recipe

def compute_metrics(predictions, gold_labels):
    # predictions and gold_labels are binary (0/1) or probabilities
    tp = sum(p and g for p, g in zip(predictions, gold_labels))
    fp = sum(p and not g for p, g in zip(predictions, gold_labels))
    fn = sum(not p and g for p, g in zip(predictions, gold_labels))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    auc = roc_auc_score(gold_labels, predictions)
    return {'AUC': auc, 'Precision': precision, 'Recall': recall, 'F1': f1}

Common pitfalls

• Confusing biased (MNAR) training/evaluation data with unbiased (MAR) golden data, leading to overoptimistic metric scores that do not reflect true user preference.
• Using sparse down-funnel signals like 'saves' for model training or direct evaluation instead of debiasing the more prevalent click data, which causes instability.
• Failing to binarize explicit 1-5 ratings into ≥4 vs <4 before computing Precision/Recall/F1, violating the paper's evaluation protocol.

Evidence (verbatim from paper)

Our method (E5) shows the lowest drift from E1 in AUC and F1-score, which balances recall and precision, compared to all baselines.

Citation

@misc{khatami2025robust,
  title={Towards Robust Offline Evaluation: A Causal and Information Theoretic Framework for Debiasing Ranking Systems},
  author={Khatami et al. (2025)},
  year={2025},
  note={arXiv:2504.03997}
}

• arXiv: 2504.03997