Autoctr Ctr Eval

autoctr-ctr-eval

Towards Automated Neural Interaction Discovery for Click-Through Rate Prediction — Qingquan Song et al. (arXiv:2007.06434, 2020)

What this evaluates

Evaluates automated neural architecture search for click-through rate prediction on heterogeneous tabular data. It measures how well discovered architectures predict user clicks compared to human-crafted models, and tests their transferability across different datasets.

Datasets

• Criteo — total ?; splits: train (-1), val (-1), test (-1)
• Avazu — total ?; splits: train (-1), val (-1), test (-1)
• KDD Cup — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• logloss (primary) — range: other
  • Negative log-likelihood of the predicted click probability for the binary outcome. Lower values indicate better calibration and accuracy.
• AUC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the model's ability to rank positive (click) instances higher than negative ones.

Input / output format

Input: Heterogeneous tabular features (sparse categorical and dense numerical) representing user-item interactions, processed according to standard CTR preprocessing pipelines.

Output: Predicted click probability or ranking score per instance.

Scoring recipe

import numpy as np
from sklearn.metrics import log_loss, roc_auc_score

def compute_metrics(y_true, y_pred):
    ll = log_loss(y_true, y_pred)
    auc = roc_auc_score(y_true, y_pred)
    return {'logloss': ll, 'AUC': auc}

Common pitfalls

• Low-fidelity search evaluation uses a fixed 2M-row subsample (80/10/10 split) rather than the full dataset, which can mislead architecture selection.
• Warm-start embedding initialization boosts low-fidelity validation scores but causes overfitting, yielding no improvement on the full test set.
• Search efficiency is measured in GPU days, but the actual training time for the searcher is explicitly ignored as negligible.

Evidence (verbatim from paper)

We use logloss and AUC score as the core evaluation metrics. Four questions are mainly explored: Q1. How is AutoCTR comparing with other baseline searchers on both the search efficiency and effectiveness? Q2. How is the performance of the best architecture explored by the AutoCTR comparing with the state-of-the-art (SOTA) human-crafted architectures?

Citation

@misc{song2020towardsautomated,
  title={Towards Automated Neural Interaction Discovery for Click-Through Rate Prediction},
  author={Qingquan Song et al.},
  year={2020},
  note={arXiv:2007.06434}
}

• arXiv: 2007.06434