Dhen Ctr Eval

dhen-ctr-eval

DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Prediction — Buyun Zhang et al. (2022) (arXiv:2203.11014, 2022)

What this evaluates

Evaluates the effectiveness of a deep hierarchical ensemble network for large-scale click-through rate (CTR) prediction. It probes the model's ability to capture complex, non-overlapping feature interactions across multiple layers and scale efficiently on industrial-scale data.

Datasets

• Industrial in-house dataset — total ?; splits: train (-1)

Metrics

• Normalized Entropy (NE) loss (primary) — range: percent
  • Normalized Entropy loss is used to evaluate CTR prediction accuracy. The paper reports the relative difference in NE loss compared to a baseline at specific training steps (e.g., @10B examples). Lower NE indicates better performance.

Input / output format

Input: Hundreds of sparse (categorical) features and thousands of dense (numerical) features per user/ad impression instance.

Output: CTR prediction probability.

Scoring recipe

baseline_ne = compute_normalized_entropy_loss(baseline_preds, labels)
model_ne = compute_normalized_entropy_loss(model_preds, labels)
# Evaluated at fixed training steps (e.g., 10B, 20B examples)
ne_diff_pct = ((model_ne - baseline_ne) / baseline_ne) * 100
return ne_diff_pct  # Negative values indicate improvement over baseline

Common pitfalls

• Normalized Entropy is a loss metric where lower values are better, but the paper reports 'NE diff' (relative improvement), which can be misinterpreted as a higher-is-better accuracy score.
• The dataset is an in-house industrial dataset with no public splits or size information, making direct replication impossible without internal access.
• Performance is measured at fixed training steps (number of examples processed) rather than fixed epochs or wall-clock time, which affects convergence comparisons.

Evidence (verbatim from paper)

To facilitate comparison, we use DCN as baseline and use relative Normalized Entropy loss difference at different training steps (training examples) to evaluate the model performance.

Citation

@misc{zhang2022dhen,
  title={DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Prediction},
  author={Buyun Zhang et al. (2022)},
  year={2022},
  note={arXiv:2203.11014}
}

• arXiv: 2203.11014