# Ctr Welfare Eval

> Evaluates click-through rate (CTR) prediction models for their ability to maximize economic welfare in simulated and real-world ad auction settings, while also measuring standard classification performance. Use when the user wants to benchmark on Synthetic Dataset, Criteo Display Advertising Challenge, or asks about evaluating this task. Reports test-time welfare.

- Skill: `qhjqhj00/ctr-welfare-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/ctr-welfare-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/ctr-welfare-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/ctr-welfare-eval

---


# ctr-welfare-eval

> Pairwise Ranking Losses of Click-Through Rates Prediction for Welfare Maximization in Ad Auctions — Lyu et al. (2023) (arXiv:2306.01799, 2023)

## What this evaluates

Evaluates click-through rate (CTR) prediction models for their ability to maximize economic welfare in simulated and real-world ad auction settings, while also measuring standard classification performance.

## Datasets

- **Synthetic Dataset** — total 12000; splits: train (10000), test (2000)
- **Criteo Display Advertising Challenge** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `test-time welfare` **(primary)** — range: other
  - Sum of predicted CTR multiplied by CPC bid for all ads in an auction, averaged over all auctions and repeated trials.
- `AUC loss` — range: [0, 1]
  - 1 minus the Area Under the Receiver Operating Characteristic Curve (AUC).
- `logistic loss` — range: [0, 1]
  - Binary cross-entropy loss between predicted CTR probabilities and ground-truth click indicators.

## Input / output format

**Input**: Ad feature vectors (50-dimensional standard normal for synthetic, standard tabular features for Criteo) and corresponding CPC bids (synthetically generated for Criteo). Ground-truth click labels are provided for training and evaluation.

**Output**: Predicted CTR probability per ad (via sigmoid activation), used to rank ads and compute welfare, AUC, and logistic loss.

## Scoring recipe

```python
# Compute per-auction welfare
welfare = sum(pred_ctr[i] * cpc_bid[i] for i in range(num_ads))
avg_welfare = np.mean(welfares)
# Compute AUC loss
auc = roc_auc_score(y_true, pred_ctr)
auc_loss = 1.0 - auc
# Compute logistic loss
eps = 1e-7
log_loss = -np.mean(y_true * np.log(pred_ctr + eps) + (1 - y_true) * np.log(1 - pred_ctr + eps))
```

## Common pitfalls

- Criteo dataset lacks actual CPC bids, slot positions, and auction IDs, requiring synthetic bid generation and a single-slot assumption that may not reflect multi-slot auction dynamics.
- Welfare scores are highly sensitive to the generated CPC bid distribution; results may not generalize to real auction environments with different bid strategies.
- Student-teacher learning requires training a separate teacher model first, which adds computational overhead and potential calibration bias if the teacher is poorly trained.

## Evidence (verbatim from paper)

> We compare the losses based on three metrics: test-time welfare, area-under-curve (AUC) loss, and logistic loss, where AUC loss is defined as $1-\textsc{AUC}$.

## Citation

```bibtex
@misc{lyu2023pairwise,
  title={Pairwise Ranking Losses of Click-Through Rates Prediction for Welfare Maximization in Ad Auctions},
  author={Lyu et al. (2023)},
  year={2023},
  note={arXiv:2306.01799}
}
```

- arXiv: 2306.01799