Ctr Welfare Eval

ctr-welfare-eval

Pairwise Ranking Losses of Click-Through Rates Prediction for Welfare Maximization in Ad Auctions — Lyu et al. (2023) (arXiv:2306.01799, 2023)

What this evaluates

Evaluates click-through rate (CTR) prediction models for their ability to maximize economic welfare in simulated and real-world ad auction settings, while also measuring standard classification performance.

Datasets

• Synthetic Dataset — total 12000; splits: train (10000), test (2000)
• Criteo Display Advertising Challenge — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• test-time welfare (primary) — range: other
  • Sum of predicted CTR multiplied by CPC bid for all ads in an auction, averaged over all auctions and repeated trials.
• AUC loss — range: [0, 1]
  • 1 minus the Area Under the Receiver Operating Characteristic Curve (AUC).
• logistic loss — range: [0, 1]
  • Binary cross-entropy loss between predicted CTR probabilities and ground-truth click indicators.

Input / output format

Input: Ad feature vectors (50-dimensional standard normal for synthetic, standard tabular features for Criteo) and corresponding CPC bids (synthetically generated for Criteo). Ground-truth click labels are provided for training and evaluation.

Output: Predicted CTR probability per ad (via sigmoid activation), used to rank ads and compute welfare, AUC, and logistic loss.

Scoring recipe

# Compute per-auction welfare
welfare = sum(pred_ctr[i] * cpc_bid[i] for i in range(num_ads))
avg_welfare = np.mean(welfares)
# Compute AUC loss
auc = roc_auc_score(y_true, pred_ctr)
auc_loss = 1.0 - auc
# Compute logistic loss
eps = 1e-7
log_loss = -np.mean(y_true * np.log(pred_ctr + eps) + (1 - y_true) * np.log(1 - pred_ctr + eps))

Common pitfalls

• Criteo dataset lacks actual CPC bids, slot positions, and auction IDs, requiring synthetic bid generation and a single-slot assumption that may not reflect multi-slot auction dynamics.
• Welfare scores are highly sensitive to the generated CPC bid distribution; results may not generalize to real auction environments with different bid strategies.
• Student-teacher learning requires training a separate teacher model first, which adds computational overhead and potential calibration bias if the teacher is poorly trained.

Evidence (verbatim from paper)

We compare the losses based on three metrics: test-time welfare, area-under-curve (AUC) loss, and logistic loss, where AUC loss is defined as $1-\textsc{AUC}$.

Citation

@misc{lyu2023pairwise,
  title={Pairwise Ranking Losses of Click-Through Rates Prediction for Welfare Maximization in Ad Auctions},
  author={Lyu et al. (2023)},
  year={2023},
  note={arXiv:2306.01799}
}

• arXiv: 2306.01799