Taobao Ctr Prediction Eval

taobao-ctr-prediction-eval

CTR Prediction on Alibaba's Taobao Advertising Dataset Using Traditional and Deep Learning Models — Yang et al. (2025) (arXiv:2511.21963, 2025)

What this evaluates

This benchmark evaluates the ability of machine learning models to predict click-through rates (CTR) for advertisements on a large-scale e-commerce platform. It specifically probes how well models capture static user-ad interactions versus dynamic, temporal user behavior sequences to forecast future clicks.

Datasets

• Alibaba's Taobao Advertising Dataset — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve, measuring the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance.
• Log Loss — range: [0, ∞)
  • Binary cross-entropy loss measuring the divergence between predicted probabilities and actual binary labels.
• PR AUC — range: [0, 1]
  • Area Under the Precision-Recall Curve, evaluating model performance on imbalanced datasets by focusing on the positive class.

Input / output format

Input: User and ad features including static demographics, ad attributes, and temporal behavioral sequences encoded via embedding layers.

Output: A single probability score representing the predicted likelihood of a user clicking on the ad.

Scoring recipe

def compute_auc(y_true, y_pred):
    from sklearn.metrics import roc_auc_score
    return roc_auc_score(y_true, y_pred)

def compute_logloss(y_true, y_pred):
    from sklearn.metrics import log_loss
    return log_loss(y_true, y_pred)

def compute_prauc(y_true, y_pred):
    from sklearn.metrics import average_precision_score
    return average_precision_score(y_true, y_pred)

Common pitfalls

• Class imbalance in click data requires careful sampling or weighting; ignoring it skews AUC/PR AUC.
• Temporal behavior sequences must be strictly ordered and padded/truncated consistently to avoid data leakage.
• Attention weights in the Transformer model indicate feature importance but do not imply causal relationships.

Evidence (verbatim from paper)

Evaluation metrics include AUC, Log Loss, and PR AUC, reflecting ranking accuracy, prediction calibration, and sensitivity to rare clicks.

Citation

@misc{yang2025ctrprediction,
  title={CTR Prediction on Alibaba's Taobao Advertising Dataset Using Traditional and Deep Learning Models},
  author={Yang et al. (2025)},
  year={2025},
  note={arXiv:2511.21963}
}

• arXiv: 2511.21963