qhjqhj00 esmm-cvr-eval
An Analysis Of Entire Space Multi-Task Models For Post-Click Conversion Prediction — Conor O'Brien et al. (2021) (arXiv:2108.13475, 2021)
What this evaluates
Evaluates multi-task learning architectures for post-click conversion rate (CVR) prediction in online advertising, specifically testing how parameter sharing and entire-space training mitigate data sparsity and bias.
Datasets
- Unspecified industry advertising dataset — total ?; splits: test (-1)
Metrics
Performance(primary) — range: relative (baseline IP = 1.0)- Normalized by the mean score of the IP baseline model (IP = 1.0). Calculated across at least 10 independent runs and reported as mean ± SEM.
Input / output format
Input: Not specified in the provided text.
Output: Not specified in the provided text.
Scoring recipe
scores = []
for run in range(10):
model_preds = predict(model, test_data)
baseline_preds = predict(IP_model, test_data)
model_score = compute_metric(model_preds, gold)
baseline_score = compute_metric(baseline_preds, gold)
scores.append(model_score / baseline_score)
mean_val = mean(scores)
sem = std(scores) / sqrt(len(scores))
return mean_val, sem
Common pitfalls
- Hyperparameter tuning significantly impacts ESMM performance more than other models, requiring careful search.
- Data bias requires explicit weighting (e.g., ESMM-NS) or parameter sharing to avoid learning noise from sparse positive install labels.
- Statistical significance must be verified via 2-sided t-tests across ≥10 independent runs due to high variance.
Evidence (verbatim from paper)
Figure 3 gives a summary of the key results from our experiments. They provide clear evidence that a meaningful decomposition of the prediction task has clear benefits, shown by ESP performing 2% worse than IP. ... The standard error for each model's performance difference (against IP mean) is calculated across at least 10 runs. The results show that there are multiple mechanisms for inducing positive transfer, but that hard parameter sharing alone (IPSP) may be optimal. Better than column indicates models that this model outperforms p < 0.01, 2-sided t-test.
Citation
@misc{obrien2021esmm,
title={An Analysis Of Entire Space Multi-Task Models For Post-Click Conversion Prediction},
author={Conor O'Brien et al. (2021)},
year={2021},
note={arXiv:2108.13475}
}
- arXiv: 2108.13475