Cvr Ctcvr Estimation Eval

cvr-ctcvr-estimation-eval

Entire Space Counterfactual Learning: Tuning, Analytical Properties and Industrial Applications — Hao Wang et al. (2022) (arXiv:2210.11039, 2022)

What this evaluates

Evaluates the ranking performance of models for click-through rate (CTR) and post-click conversion rate (CVR) estimation in recommendation systems. It probes the model's ability to correctly rank items by their predicted probability of conversion, while mitigating sample selection bias and false independence assumptions between clicks and conversions.

Datasets

• Industrial Benchmark — total ?; splits: train (-1), val (-1), test (-1)
• Ali-CCP — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve. Measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance.
• Recall — range: [0, 1]
  • True positive rate at the best threshold on the precision-recall curve.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall at the best threshold on the precision-recall curve.
• KS — range: [0, 1]
  • Kolmogorov-Smirnov statistic measuring the maximum absolute difference between the cumulative distribution functions of positive and negative predictions.

Input / output format

Input: Single-valued categorical fields representing user, item, and context features.

Output: Predicted probability scores for CTR and CVR/CTCVR.

Scoring recipe

def compute_auc(y_true, y_pred):
    order = np.argsort(-y_pred)
    y_true = y_true[order]
    tp = np.cumsum(y_true)
    fp = np.cumsum(1 - y_true)
    tpr = tp / tp[-1]
    fpr = fp / fp[-1]
    return np.trapz(tpr, fpr)

def compute_recall_f1(y_true, y_pred, threshold):
    y_pred_bin = (y_pred >= threshold).astype(int)
    tp = np.sum((y_pred_bin == 1) & (y_true == 1))
    fn = np.sum((y_pred_bin == 0) & (y_true == 1))
    fp = np.sum((y_pred_bin == 1) & (y_true == 0))
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return recall, f1

Common pitfalls

• Negative samples are heavily downsampled in training (100:10:1 ratio), which distorts class priors and requires careful threshold selection for Recall/F1.
• KS metric is reported but measures distribution shift rather than ranking quality, making direct comparison with AUC misleading.
• Online A/B test metrics (UV-CVR, # Order) are business-level aggregates over days, not instance-level model scores, and can fluctuate daily.

Evidence (verbatim from paper)

We mainly use the area under roc (AUC) metric to compare the ranking performance of models. Nevertheless, it only reveals the ranking performance that is averaged at all thresholds. To better understand the performance of each model, we also report the model performance for the best threshold on the receiver operating characteristic curve and the recall and F1 scores for the best threshold on the precision-recall curve, respectively.

Citation

@misc{wang2022entire,
  title={Entire Space Counterfactual Learning: Tuning, Analytical Properties and Industrial Applications},
  author={Hao Wang et al. (2022)},
  year={2022},
  note={arXiv:2210.11039}
}

• arXiv: 2210.11039