Deepfm Ctr Eval

deepfm-ctr-eval

DeepFM: A Factorization-Machine based Neural Network for CTR Prediction — Guo et al. (2017) (arXiv:1703.04247, 2017)

What this evaluates

Evaluates click-through rate (CTR) prediction models by measuring their ability to correctly rank clicked versus non-clicked instances and output calibrated click probabilities.

Datasets

• Criteo Dataset — total 45000000; splits: train (-1), test (-1)
• Company Dataset* — total 1000000000; splits: train (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve; measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance.
• Logloss — range: other
  • Binary cross-entropy loss: -mean(y_true * log(y_pred) + (1 - y_true) * log(1 - y_pred)). Lower values indicate better probability calibration.

Input / output format

Input: A feature vector containing 13 continuous features and 26 categorical features (or app/user/context features for Company*), paired with a binary label indicating whether the user clicked.

Output: A single scalar probability score representing the predicted likelihood of a click.

Scoring recipe

def compute_auc(y_true, y_pred):
    pos = y_true == 1
    neg = y_true == 0
    return np.mean(y_pred[pos][:, None] > y_pred[neg][None, :])

def compute_logloss(y_true, y_pred):
    eps = 1e-15
    y_pred = np.clip(y_pred, eps, 1 - eps)
    return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))

Common pitfalls

• The Criteo dataset uses a random 90/10 split rather than a temporal split, which can cause data leakage if the model is evaluated on future data in practice.
• Logloss is reported to 5 decimal places, but absolute differences of ~0.0001 may lack statistical significance without variance or confidence intervals.
• Efficiency is measured as a relative ratio to LR training time, not absolute wall-clock seconds, making cross-hardware comparisons unreliable.

Evidence (verbatim from paper)

We use two evaluation metrics in our experiments: AUC (Area Under ROC) and Logloss (cross entropy). The performance for CTR prediction of different models on Criteo dataset and Company* dataset is shown in Table 2, where we have the following observations: DeepFM outperforms LR by 0.86% and 4.18% in terms of AUC (1.15% and 5.60% in terms of Logloss) on Company* and Criteo datasets.

Citation

@misc{guo2017deepfm,
  title={DeepFM: A Factorization-Machine based Neural Network for CTR Prediction},
  author={Guo et al. (2017)},
  year={2017},
  note={arXiv:1703.04247}
}

• arXiv: 1703.04247