Criteo Dlrm Eval

criteo-dlrm-eval

Random Offset Block Embedding Array (ROBE) for CriteoTB Benchmark MLPerf DLRM Model : 1000$\times$ Compression and 3.1$\times$ Faster Inference — Desai et al. (2021) (arXiv:2108.02191, 2021)

What this evaluates

Evaluates the predictive accuracy and inference efficiency of deep learning recommendation models (DLRM) with compressed embedding tables on large-scale advertising click-through rate datasets. It measures Area Under the ROC Curve (AUC) to assess model quality and samples per second to quantify inference throughput under memory-constrained conditions.

Datasets

• CriteoTB — total ?; splits: train (-1), val (-1), test (-1)
• Criteo Kaggle — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Computed over predicted click-through rate probabilities and binary ground-truth labels.
• Throughput — range: other
  • Inference speed measured in samples processed per second. Includes data transfer from RAM to GPU and hash computation.

Input / output format

Input: Batch of 13 integer features and 26 categorical features (tokens) representing user/ad interactions.

Output: Binary click/no-click label (0 or 1) or predicted probability score.

Scoring recipe

def compute_auc(y_true, y_pred):
    # y_true: binary ground truth labels
    # y_pred: predicted probabilities
    fpr, tpr, _ = roc_curve(y_true, y_pred)
    return auc(fpr, tpr)

Common pitfalls

• The MLPerf CriteoTB benchmark strictly requires achieving an AUC of at least 0.8025 to be considered valid; simply reporting AUC without meeting this threshold may not be comparable.
• Inference throughput measurements include data transfer from RAM to GPU and hash computation, not just the forward pass, which can skew comparisons if not standardized.
• Training time comparisons are confounded by the number of epochs required to reach target AUC, as compressed models often need more epochs despite smaller memory footprint.

Evidence (verbatim from paper)

We will use the same quality metric of 0.8025 AUC as prescribed in MLPerf settings for CriteoTB dataset to evaluate ROBE-Z.

Citation

@misc{desai2021robe,
  title={Random Offset Block Embedding Array (ROBE) for CriteoTB Benchmark MLPerf DLRM Model : 1000$\times$ Compression and 3.1$\times$ Faster Inference},
  author={Desai et al. (2021)},
  year={2021},
  note={arXiv:2108.02191}
}

• arXiv: 2108.02191