Esmm Cvr Ctcvr Eval

esmm-cvr-ctcvr-eval

Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate — Xiao Ma et al. (2018) (arXiv:1804.07931, 2018)

What this evaluates

Evaluates a model's ability to estimate post-click conversion rate (CVR) and post-click-and-conversion rate (CTCVR) in recommendation systems. It specifically probes how well the model handles sample selection bias and data sparsity by comparing performance on clicked-only impressions versus the entire impression space.

Datasets

• Public Dataset — total 84000000; splits: train (42000000), test (42000000)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve. It measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance based on the predicted probability scores.

Input / output format

Input: Impression-level records containing user ID, item ID, and contextual features, labeled with binary click (0/1) and conversion (0/1) outcomes.

Output: A single probability score per impression representing the predicted likelihood of conversion (pCVR) or click-and-conversion (pCTCVR).

Scoring recipe

def compute_auc(y_true, y_pred):
    # y_true: binary labels (0 or 1)
    # y_pred: predicted probabilities
    sorted_indices = np.argsort(-y_pred)
    y_true_sorted = y_true[sorted_indices]
    tp = np.cumsum(y_true_sorted)
    fp = np.cumsum(1 - y_true_sorted)
    tpr = tp / tp[-1]
    fpr = fp / fp[-1]
    return np.trapz(tpr, fpr)

Common pitfalls

• The CVR task is evaluated exclusively on clicked impressions (biased subset), while CTCVR is evaluated on all impressions (entire space). Confusing these splits invalidates the sample selection bias analysis.
• For CTCVR evaluation, baseline models must use a fixed, independently trained pCTR network rather than their own predicted pCTR to isolate the CVR component.
• The dataset split is strictly time-based (first half train, second half test). Random shuffling breaks the temporal distribution and leads to data leakage.

Evidence (verbatim from paper)

Both of the two tasks split the first 1/2 data in the time sequence to be training set while the rest to be test set. Area under the ROC curve (AUC) is adopted as performance metrics. All experiments are repeated 10 times and averaged results are reported.

Citation

@misc{ma2018entire,
  title={Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate},
  author={Xiao Ma et al. (2018)},
  year={2018},
  note={arXiv:1804.07931}
}

• arXiv: 1804.07931