Deepctr Ctr Prediction Eval

deepctr-ctr-prediction-eval

Deep CTR Prediction in Display Advertising — Chen et al. (2016) (arXiv:1609.06018, 2016)

What this evaluates

Evaluates a model's ability to predict click-through rates for display advertisements by combining raw image pixels with contextual features. It probes the model's capacity to learn high-level visual semantics and complex nonlinear interactions for ranking and probability calibration in a highly imbalanced, real-world advertising setting.

Datasets

• Commercial Display Ad Dataset (2015) — total 59000000; splits: train (50000000), test (9000000), test_new_images (3090)

Metrics

• relative AUC (primary) — range: percent
  • Relative improvement over a logistic regression baseline: ((AUC(method) - 0.5) / (AUC(lr_basic) - 0.5) - 1) * 100%. Measures ranking quality improvement.
• relative Logloss — range: percent
  • Relative improvement over logistic regression baseline, reported as a percentage. Lower values indicate better probability calibration.

Input / output format

Input: RGB image (112x112) concatenated with a one-hot encoded basic feature vector (dimension 153,231) containing ad zone, ad group, ad target, ad category, and user demographics.

Output: Predicted click probability (scalar between 0 and 1).

Scoring recipe

predictions: array of predicted click probabilities
labels: array of binary click labels (1=click, 0=no-click)
auc = roc_auc_score(labels, predictions)
logloss = log_loss(labels, predictions)
auc_base = roc_auc_score(labels, predictions_baseline)
logloss_base = log_loss(labels, predictions_baseline)
rel_auc = ((auc - 0.5) / (auc_base - 0.5) - 1) * 100
rel_logloss = ((logloss - logloss_base) / logloss_base) * 100
return rel_auc, rel_logloss

Common pitfalls

• Metrics are reported as relative percentages compared to a logistic regression baseline, not absolute values.
• The dataset is proprietary and contains a severe class imbalance (1:30 positive:negative) that is explicitly not subsampled.
• The test set includes a 'new images' subset (3,090 samples) where ad groups were never seen during training, requiring careful handling of feature availability.

Evidence (verbatim from paper)

We use two popular metrics to evaluate the experiment result, Logloss and the area under receiver operator curve (AUC). Logloss can quantify the accuracy of the predicted click probability. AUC measures the ranking quality of the prediction. Our dataset comes from a real commercial platform, so both of these metrics use relative numbers comparing with lr basic. Since the AUC value is always larger than 0.5, we remove this constant part (0.5) from the AUC value and then compute the relative numbers as in [30]: relative AUC = ((AUC(method) - 0.5) / (AUC(lr_basic) - 0.5) - 1) * 100%

Citation

@misc{chen2016deepctr,
  title={Deep CTR Prediction in Display Advertising},
  author={Chen et al. (2016)},
  year={2016},
  note={arXiv:1609.06018}
}

• arXiv: 1609.06018