Calicalcausalrank Eval

calicalcausalrank-eval

CaliCausalRank: Calibrated Multi-Objective Ad Ranking with Robust Counterfactual Utility Optimization — Yang et al. (2026) (arXiv:2602.18786, 2026)

What this evaluates

Evaluates multi-objective ad ranking models on click-through rate (CTR) and conversion rate (CVR) prediction tasks, focusing on ranking quality, score calibration, and counterfactual utility estimation under position and selection bias.

Datasets

• Criteo — total 45000000; splits: train (-1), test (-1)
• Avazu — total 40000000; splits: train (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring pairwise ranking quality between positive and negative samples.
• NDCG@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10, measuring the quality of the top-k relevance ordering relative to an ideal ranking.
• Expected Calibration Error (ECE) — range: [0, 1]
  • Weighted average absolute difference between predicted probabilities and actual positive frequencies across K=20 calibration buckets.
• Utility@10 — range: other
  • Counterfactual utility estimate for the top-10 ranked items, computed using variance-reduced off-policy estimators to correct for position and selection bias.

Input / output format

Input: Ad impression features including 13 numerical and 26 categorical features (Criteo) or device and temporal features (Avazu), with binary labels for click (CTR) and synthetic/real conversion (CVR).

Output: Predicted probability scores for CTR and CVR, used to rank ads and compute ranking, calibration, and utility metrics.

Scoring recipe

def compute_metrics(y_true, y_pred, topk=10, buckets=20):
    auc = roc_auc_score(y_true, y_pred)
    ndcg = ndcg_at_k(y_true, y_pred, k=topk)
    bin_edges = np.linspace(0, 1, buckets + 1)
    ece = 0.0
    for i in range(buckets):
        mask = (y_pred >= bin_edges[i]) & (y_pred < bin_edges[i+1])
        if mask.sum() > 0:
            ece += mask.sum() * abs(y_true[mask].mean() - y_pred[mask].mean())
    ece /= len(y_true)
    utility = counterfactual_utility_estimate(y_true, y_pred, topk)
    return {'AUC': auc, 'NDCG@10': ndcg, 'ECE': ece, 'Utility@10': utility}

Common pitfalls

• CVR labels are synthetically generated only for the clicked subset (~3% of samples) with a fixed 10% base probability, which may not reflect real-world conversion distributions or introduce label leakage if not handled carefully.
• Utility@10 relies on counterfactual/off-policy estimation to correct for position and selection bias, meaning scores are not directly comparable to standard online A/B test metrics without proper variance reduction.

Evidence (verbatim from paper)

We report: AUC for ranking quality, NDCG@10 for top-k performance, Expected Calibration Error (ECE) for calibration, and Utility@10 computed using counterfactual evaluation.

Citation

@misc{yang2026calicausalrank,
  title={CaliCausalRank: Calibrated Multi-Objective Ad Ranking with Robust Counterfactual Utility Optimization},
  author={Yang et al. (2026)},
  year={2026},
  note={arXiv:2602.18786}
}

• arXiv: 2602.18786