Online Ctr Eval

online-ctr-eval

Online Interaction Detection for Click-Through Rate Prediction — Lin et al. (2021) (arXiv:2106.15400, 2021)

What this evaluates

This protocol evaluates an online feature interaction detection method integrated into click-through rate (CTR) prediction models. It measures predictive accuracy on streaming ad click data using chronological splits to simulate real-time recommendation scenarios.

Datasets

• Avazu — total 40428967; splits: D1-D4 (-1), D5 (-1), D6-D10 (-1)
• Criteo — total 45840617; splits: D1-D4 (-1), D5 (-1), D6-D10 (-1)
• Taobao — total 26557961; splits: D1-D4 (-1), D5 (-1), D6-D10 (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve. Measures the probability that a randomly chosen positive instance ranks higher than a randomly chosen negative instance.
• logloss (primary) — range: other
  • Binary cross-entropy loss, defined as $-\frac{1}{N}\sum_{i=1}^N [y_i \log(\hat{y}_i) + (1-y_i) \log(1-\hat{y}_i)]$. Lower values indicate better calibrated probability estimates.

Input / output format

Input: Tabular records containing categorical and numerical features representing user context and ad properties, ordered chronologically.

Output: A single continuous probability value indicating the likelihood of a user clicking the displayed ad.

Scoring recipe

import numpy as np
from sklearn.metrics import roc_auc_score

def compute_auc(y_true, y_pred):
    return roc_auc_score(y_true, y_pred)

def compute_logloss(y_true, y_pred):
    eps = 1e-15
    y_pred = np.clip(y_pred, eps, 1 - eps)
    return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))

Common pitfalls

• Chronological splitting is mandatory; random shuffling breaks the streaming simulation and inflates performance.
• In CTR benchmarks, improvements as small as 0.001 in AUC or logloss are considered practically significant.
• Unfreezing the base model during online fine-tuning often causes severe overfitting on validation sets that poorly represent future data.

Evidence (verbatim from paper)

Two common metrics for CTR prediction are adopted to evaluate the models, namely AUC (Area Under ROC) and logloss (cross-entropy). It is noticeable that a small increase of AUC or a slight decrease of logloss at 0.001-level is regarded significant for CTR prediction task, according to existing works [[3], [4], [8], [15]].

Citation

@misc{lin2021online,
  title={Online Interaction Detection for Click-Through Rate Prediction},
  author={Lin et al. (2021)},
  year={2021},
  note={arXiv:2106.15400}
}

• arXiv: 2106.15400