Fibinet Ctr Eval

fibinet-ctr-eval

FiBiNET: Combining Feature Importance and Bilinear feature Interaction for Click-Through Rate Prediction — Huang et al. (2019) (arXiv:1905.09433, 2019)

What this evaluates

Evaluates the ability of shallow and deep learning models to predict click-through rates (CTR) on large-scale ad impression datasets. It probes how well architectures can model high-order feature interactions and dynamically weight feature importance using bilinear functions and Squeeze-Excitation networks.

Datasets

• Criteo — total 45000000; splits: train (-1), test (-1); repo http://labs.criteo.com/downloads/download-terabyte-click-logs/
• Avazu — total 40000000; splits: train (-1), test (-1); repo http://www.kaggle.com/c/avazu-ctr-prediction

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve. Measures the model's ability to distinguish between positive and negative instances across all classification thresholds. Larger values indicate better performance.
• Log loss — range: [0, inf)
  • Logarithmic loss measuring the distance between predicted probability distributions and true binary labels. Lower values indicate better calibration and performance.

Input / output format

Input: Sparse categorical and continuous feature vectors representing ad impressions (e.g., 26 categorical + 13 continuous fields for Criteo; 24 fields for Avazu).

Output: Predicted probability of a click (binary classification).

Scoring recipe

def compute_auc(y_true, y_pred):
    return roc_auc_score(y_true, y_pred)
def compute_logloss(y_true, y_pred):
    y_pred = np.clip(y_pred, 1e-15, 1 - 1e-15)
    return -np.mean(y_true * np.log(y_pred) + (1 - y_true) * np.log(1 - y_pred))

Common pitfalls

• An improvement of 1‰ (0.001) in AUC is considered practically significant for CTR prediction due to large user bases.
• Log loss is minimized (lower is better), whereas AUC is maximized (higher is better).
• Embedding dimensions and DNN hyperparameters are dataset-specific and require careful tuning to avoid overfitting or optimization difficulties.

Evidence (verbatim from paper)

In our experiment, we adopt two metrics: AUC(Area Under ROC) and Log loss. AUC: Area under ROC curve is a widely used metric in evaluating classification problems. Besides, some work validates AUC as a good measurement in CTR prediction*(Graepel et al., 2010)*. AUC is insensitive to the classification threshold and the positive ratio. The upper bound of AUC is 1, and the larger the better. Log loss: Log loss is widely used metric in binary classification, measuring the distance between two distributions. The lower bound of log loss is 0, indicating the two distributions perfectly match, and a smaller value indicates better performance.

Citation

@misc{huang2019fibinet,
  title={FiBiNET: Combining Feature Importance and Bilinear feature Interaction for Click-Through Rate Prediction},
  author={Huang et al. (2019)},
  year={2019},
  note={arXiv:1905.09433}
}

• arXiv: 1905.09433