Persia Ctr Eval

persia-ctr-eval

Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters — Lian et al. (2021) (arXiv:2111.05897, 2021)

What this evaluates

Evaluates the training efficiency, scalability, and convergence of a hybrid deep learning recommender system against baselines on click-through rate (CTR) prediction tasks. It measures end-to-end training time to reach target AUC, final test AUC for statistical efficiency, and training throughput across varying model scales up to 100 trillion parameters.

Datasets

• Taobao-Ad — total 26000000; splits: train (20800000), test (5200000)
• Avazu-Ad — total 32000000; splits: train (25600000), test (6400000)
• Criteo-Ad — total 44000000; splits: train (35200000), test (8800000)
• Kwai-Video — total 3000000000; splits: train (2550000000), test (450000000)
• Criteo-Syn — total ?; splits: train (-1), test (-1)

Metrics

• test AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
• training sample throughput — range: other
  • Number of training samples processed per second during end-to-end training.

Input / output format

Input: Tabular features containing high-cardinality categorical IDs and dense numerical fields, fed into a 5-layer FFNN (4096, 2048, 1024, 512, 256) with embedding layers.

Output: Binary click/behavior probability (CTR prediction).

Scoring recipe

def compute_auc(y_true, y_pred):
    sorted_idx = np.argsort(y_pred)[::-1]
    y_true_s = y_true[sorted_idx]
    tpr = np.cumsum(y_true_s) / np.sum(y_true)
    fpr = np.cumsum(1 - y_true_s) / np.sum(1 - y_true)
    return np.trapz(tpr, fpr)

Common pitfalls

• Confusing end-to-end training time to reach a target AUC with final convergence AUC.
• Criteo-Syn is a synthetic extension of Criteo-Ad for capacity testing, not a standard public benchmark.
• Throughput comparisons span different hardware clusters (Kwai internal vs. Google Cloud) without normalization.

Evidence (verbatim from paper)

We report test AUC to evaluate convergence. ... We first compare the end-to-end training time that each system needs to achieve a given AUC, using the three open source benchmarks over a heterogeneous cluster with 8 GPU workers. ... For the three open source advertisement CTR benchmarks, we include 80% of the records as training set and the rest 20% of the records as test set, we consider a fully connected feed forward neural network (FFNN) as the deep learning model with five hidden layer dimensions of 4096, 2048, 1024, 512 and 256. For the Kwai production microvideo recommendation task, 85% of the data are included in the training set while the rest 15% are considered as the test set

Citation

@misc{lian2021persia,
  title={Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters},
  author={Lian et al. (2021)},
  year={2021},
  note={arXiv:2111.05897}
}

• arXiv: 2111.05897