qhjqhj00 rec-auc-eval
On the Embedding Collapse when Scaling up Recommendation Models — Guo et al. (2023) (arXiv:2310.04400, 2023)
What this evaluates
Evaluates the predictive performance of recommendation models on large-scale click-through rate datasets. It specifically probes how model scalability and embedding size affect ranking quality, revealing the phenomenon of embedding collapse when scaling up feature interactions.
Datasets
- Criteo — total 45800000; splits: train (36640000), val (4580000), test (4580000)
- Avazu — total 40400000; splits: train (32320000), val (4040000), test (4040000)
Metrics
AUC(primary) — range: [0, 1]- Area Under the Receiver Operating Characteristic Curve. Measures the probability that a randomly chosen positive instance (click) is ranked higher than a randomly chosen negative instance (non-click).
Input / output format
Input: Tabular data with categorical fields representing user and item features for click-through rate prediction.
Output: Probability score indicating the likelihood of a click.
Scoring recipe
def compute_auc(y_true, y_pred):
from sklearn.metrics import roc_auc_score
return roc_auc_score(y_true, y_pred)
Common pitfalls
- Scaling embedding dimensions (2x to 10x) does not improve AUC due to embedding collapse; performance plateaus or degrades.
- Early stopping is applied on validation AUC with a patience of 3, not on training loss or test performance.
- Experiments are repeated 3 times with different random initializations, but dataset splits use a fixed random seed (0).
Evidence (verbatim from paper)
For all experiments, we split the dataset into 8:1:1 for training/validation/test with random seed 0. ... All experiments use early stopping on validation AUC with patience 3. We repeat each experiment for 3 times with different random initialization.
Citation
@misc{guo2023embeddingcollapse,
title={On the Embedding Collapse when Scaling up Recommendation Models},
author={Guo et al. (2023)},
year={2023},
note={arXiv:2310.04400}
}
- arXiv: 2310.04400