Ctr Prediction Eval

ctr-prediction-eval

Structured Semantic Model supported Deep Neural Network for Click-Through Rate Prediction — Niu et al. (2018) (arXiv:1812.01353, 2018)

What this evaluates

Evaluates the ability of deep learning models to predict click-through rates (CTR) from sparse, high-dimensional categorical features in advertising and recommendation scenarios. It probes how well models capture multi-scale semantic interactions and handle large-scale, imbalanced binary classification tasks typical of real-world ad systems.

Datasets

• Avazu — total 40000000; splits: train (-1), test (-1)
• MovieLens — total 20000263; splits: train (-1), test (-1)
• Weibo — total 1100000000; splits: train (1000000000), test (100000000)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve. Measures the ranking quality of predicted CTR scores against binary click/no-click labels across all intra-user and inter-user pairs.
• RelaImpr — range: percent
  • Relative improvement over a baseline model, calculated as ((AUC_model - 0.5) / (AUC_base - 0.5) - 1) * 100%. Used to normalize performance gains across datasets with different difficulty levels.

Input / output format

Input: Sparse categorical feature vectors (22–31 fields covering user profile, ad information, and context) with a binary click/no-click label.

Output: A single continuous probability score (predicted CTR) per sample.

Scoring recipe

def compute_auc(labels, preds):
    total_pos = sum(labels)
    total_neg = len(labels) - total_pos
    sorted_idx = np.argsort(preds)[::-1]
    labels_sorted = np.array(labels)[sorted_idx]
    tp = fp = 0
    auc = 0.0
    prev_fpr = prev_tpr = 0.0
    for y in labels_sorted:
        if y == 1: tp += 1
        else: fp += 1
        tpr = tp / total_pos
        fpr = fp / total_neg
        auc += (fpr - prev_fpr) * (tpr + prev_tpr) / 2
        prev_fpr, prev_tpr = fpr, tpr
    return auc

def compute_rela_impr(auc_model, auc_base):
    return ((auc_model - 0.5) / (auc_base - 0.5) - 1) * 100

Common pitfalls

• RelaImpr becomes numerically unstable or undefined if the baseline AUC is close to 0.5, as the denominator approaches zero.
• MovieLens requires manual binary thresholding (rating > 3) to fit the CTR task, which differs from standard rating prediction benchmarks and can alter class distribution.
• Weibo dataset sizes are approximate ('about 1 billion'), and exact temporal split boundaries may vary across implementations, affecting reproducibility.

Evidence (verbatim from paper)

AUC is one of the most popular evaluation metrics for CTR prediction which measures the goodness of order by ranking all the ads with predicted CTR, including intra-user and inter-user orders. We adopt RelaImpr introduced in Yan et al. (2014) to measure relative improvement over models It is defined as follows: RelaImpr=((AUC(measured model)-0.5)/(AUC(base model)-0.5)-1)×100%.

Citation

@misc{niu2018structured,
  title={Structured Semantic Model supported Deep Neural Network for Click-Through Rate Prediction},
  author={Niu et al. (2018)},
  year={2018},
  note={arXiv:1812.01353}
}

• arXiv: 1812.01353