Dcmt Cvr Eval

dcmt-cvr-eval

DCMT: A Direct Entire-Space Causal Multi-Task Framework for Post-Click Conversion Estimation — Feng Zhu et al. (2023) (arXiv:2302.06141, 2023)

What this evaluates

Evaluates the ability of multi-task learning models to predict post-click conversion rate (CVR) and click-through conversion rate (CTCVR) while mitigating selection bias in recommendation and search systems. It probes whether causal debiasing mechanisms improve ranking quality on both clicked and unclicked items across diverse e-commerce and industrial datasets.

Datasets

• Ali-CCP — total ?; splits: (unstated)
• Ali-Express (AE-ES) — total ?; splits: (unstated)
• Ali-Express (AE-FR) — total ?; splits: (unstated)
• Ali-Express (AE-NL) — total ?; splits: (unstated)
• Ali-Express (AE-US) — total ?; splits: (unstated)
• Alipay Search — total ?; splits: train (7), val (1), test (2)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve. Computed as the integral of the true positive rate versus false positive rate across all classification thresholds. Ranges from 0 to 1, where 1 indicates perfect ranking.
• PV-CTR — range: percent
  • Click-through rate per page view, calculated as total clicks divided by total page views in the online experiment.
• PV-CVR — range: percent
  • Conversion rate per page view, calculated as total conversions divided by total page views.
• Top-5 PV-CVR — range: percent
  • Conversion rate per page view restricted to the top-5 displayed services on a screen.

Input / output format

Input: User features, item features, combination features, context features, and exposure features representing the clicked or non-clicked items in the recommendation list.

Output: Continuous probability scores for click (CTR) and conversion (CVR) for each candidate item.

Scoring recipe

import numpy as np
from sklearn.metrics import roc_auc_score

def compute_auc(y_true, y_pred):
    # y_true: binary labels (0/1) for click or conversion
    # y_pred: predicted probabilities
    if len(np.unique(y_true)) < 2:
        return 0.0
    return roc_auc_score(y_true, y_pred)

# Offline evaluation
auc_cvr = compute_auc(gold_cvr_labels, pred_cvr_scores)
auc_ctcvr = compute_auc(gold_ctcvr_labels, pred_ctcvr_scores)
# Average over 5 random seeds

Common pitfalls

• CVR labels are only observable for clicked items (factual space), so naive offline evaluation on clicked data severely overestimates performance due to selection bias.
• Online business metrics (PV-CTR, PV-CVR) measure system-level impact, while offline AUC measures ranking quality; improvements in AUC do not always translate to online metric gains.
• Datasets are proprietary industrial logs with encrypted features, preventing exact feature reproduction or public dataset replication without platform access.

Evidence (verbatim from paper)

The area under the ROC curve (AUC) is adopted as performance metrics in the experiments. All experiments are repeated 5 times and averaged results are reported.

Citation

@misc{zhu2023dcmt,
  title={DCMT: A Direct Entire-Space Causal Multi-Task Framework for Post-Click Conversion Estimation},
  author={Feng Zhu et al. (2023)},
  year={2023},
  note={arXiv:2302.06141}
}

• arXiv: 2302.06141