Criteo Ctr Eval

criteo-ctr-eval

The trade-offs of model size in large recommendation models : A 10000 $\times$ compressed criteo-tb DLRM model (100 GB parameters to mere 10MB) — Desai et al. (2022) (arXiv:2207.10731, 2022)

What this evaluates

Evaluates the predictive quality and system efficiency of deep learning recommendation models on click-through rate prediction. It measures how well parameter-sharing compression techniques maintain model accuracy while reducing memory footprint and improving training and inference latency.

Datasets

• criteo-kaggle — total ?; splits: train (-1), val (-1), test (-1)
• criteo-tb — total 800000000; splits: test (89000000)

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve, computed over predicted click probabilities and binary ground truth labels to measure ranking quality.

Input / output format

Input: 13 integer features and 26 categorical features per sample.

Output: Binary classification probability (click/no-click) for CTR prediction.

Scoring recipe

def compute_auc(predictions, labels):
    # predictions: array of float probabilities
    # labels: array of binary ground truth labels
    fpr, tpr, _ = roc_curve(labels, predictions)
    return auc(fpr, tpr)

Common pitfalls

• Assuming high compression ratios (e.g., 10000x) inherently degrade model quality; the protocol shows AUC remains stable up to extreme compression.
• Focusing only on convergence speed (epochs) without accounting for per-iteration latency gains; overall training time remains comparable due to faster computation per iteration.

Evidence (verbatim from paper)

Facebook MLPerf DLRM[2] model, available under Apache-2.0 license, for the criteo-tb dataset achieves the target AUC (0.8025) with the embedding memory of around 100GB. This model uses a maximum cap of 40M indices per embedding table, leading to a total of 204M embeddings.

Citation

@misc{desai2022modelsizecompression,
  title={The trade-offs of model size in large recommendation models : A 10000 $\times$ compressed criteo-tb DLRM model (100 GB parameters to mere 10MB)},
  author={Desai et al. (2022)},
  year={2022},
  note={arXiv:2207.10731}
}

• arXiv: 2207.10731