Dien Ctr Eval

dien-ctr-eval

Deep Interest Evolution Network for Click-Through Rate Prediction — Zhou et al. (2018) (arXiv:1809.03672, 2018)

What this evaluates

Evaluates a model's ability to predict click-through rates (CTR) by modeling sequential user behavior and dynamically evolving latent interests relative to a target item.

Datasets

• Amazon Books — total 603668; splits: train (-1), test (-1)
• Amazon Electronics — total 192403; splits: train (-1), test (-1)
• Industrial (Taobao) — total 7000000000; splits: train (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the probability that a randomly chosen positive instance (click) is ranked higher than a randomly chosen negative instance (non-click).

Input / output format

Input: User's historical behavior sequence (e.g., reviews or clicked items) up to time T-1, and a target item at time T.

Output: A predicted probability score indicating the likelihood of the user clicking the target item.

Scoring recipe

def compute_auc(y_true, y_pred):
    pos_scores = [s for s, y in zip(y_pred, y_true) if y == 1]
    neg_scores = [s for s, y in zip(y_pred, y_true) if y == 0]
    if not pos_scores or not neg_scores:
        return 0.0
    concordant = sum(1 for p in pos_scores for n in neg_scores if p > n)
    tied = sum(1 for p in pos_scores for n in neg_scores if p == n)
    return (concordant + 0.5 * tied) / (len(pos_scores) * len(neg_scores))

Common pitfalls

• The industrial dataset uses a strict temporal split (training on past 49 days, testing on the next day), so random shuffling will cause data leakage and inflate metrics.
• AUC is reported as mean ± std over 5 runs, but the paper does not specify the exact random seed or data partitioning strategy for the public datasets, making exact replication difficult.
• Online A/B testing metrics (CTR/eCPM gain) are relative to a specific baseline (BaseModel) and cannot be directly compared to offline AUC without knowing the baseline's absolute values.

Evidence (verbatim from paper)

We use both public and industrial datasets to verify the effect of DIEN. ... Each experiment is repeated 5 times. Table 2: Results (AUC) on public datasets ... Table 3: Results (AUC) on industrial dataset

Citation

@misc{zhou2018dien,
  title={Deep Interest Evolution Network for Click-Through Rate Prediction},
  author={Zhou et al. (2018)},
  year={2018},
  note={arXiv:1809.03672}
}

• arXiv: 1809.03672