Fcn Ctr Eval

fcn-ctr-eval

FCN: Fusing Exponential and Linear Cross Network for Click-Through Rate Prediction — Li et al. (2024) (arXiv:2407.13349, 2024)

What this evaluates

Evaluates the effectiveness, efficiency, and interpretability of explicit feature interaction models for click-through rate (CTR) prediction on large-scale, highly sparse advertising and recommendation datasets.

Datasets

• Avazu — total 40428967; splits: train/val/test (-1)
• Criteo — total 45840617; splits: train/val/test (-1)
• ML-1M — total 739012; splits: train/val/test (-1)
• KDD12 — total 141371038; splits: train/val/test (-1)
• iPinYou — total 19495974; splits: train/val/test (-1)
• KKBox — total 7377418; splits: train/val/test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative one.
• Logloss — range: other
  • Negative log-likelihood loss. Calculated as -1/N * sum(y_true * log(y_pred) + (1 - y_true) * log(1 - y_pred)). Lower values indicate better model fit.

Input / output format

Input: Sparse categorical and numerical feature vectors representing user-item interactions. Numerical features are discretized via floor(log^2(x)) for x>2, and infrequent categories are replaced with an 'OOV' token.

Output: A single scalar probability score representing the likelihood of a click.

Scoring recipe

import numpy as np
from sklearn.metrics import roc_auc_score, log_loss

def compute_metrics(y_true, y_pred):
    auc = roc_auc_score(y_true, y_pred)
    ll = log_loss(y_true, y_pred)
    return {'AUC': auc, 'Logloss': ll}

Common pitfalls

• Small absolute improvements (e.g., 0.1% in AUC or 0.001 in Logloss) are considered statistically significant and meaningful in CTR tasks.
• Dataset class imbalance can cause Logloss to plateau at similar low values (e.g., ~0.0055) across models, making AUC a more reliable differentiator.
• Parameter count does not directly correlate with training runtime or time complexity; some lightweight models have high time complexity.

Evidence (verbatim from paper)

To compare the performance, we utilize two commonly used metrics in CTR models: Logloss, AUC (Song et al., [2019]; Wang et al., [2023a]; Zhu et al., [2022b]). AUC stands for Area Under the ROC Curve, which measures the probability that a positive instance will be ranked higher than a randomly chosen negative one. Logloss is the result of the calculation of L in Equation [7]. A lower Logloss suggests a better capacity for fitting the data.

Citation

@misc{li2024fcn,
  title={FCN: Fusing Exponential and Linear Cross Network for Click-Through Rate Prediction},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2407.13349}
}

• arXiv: 2407.13349