Ctr Prediction Dhan Eval

ctr-prediction-dhan-eval

Deep Interest with Hierarchical Attention Network for Click-Through Rate Prediction — Weinan Xu et al. (2020) (arXiv:2005.12981, 2020)

What this evaluates

Evaluates a model's ability to predict the next item a user will click or review based on their historical interaction sequence. It probes hierarchical user interest modeling across multiple product dimensions and abstraction levels in recommendation systems.

Datasets

• Amazon Review (Six-Category, Kindle Shop, Electronics) — total ?; splits: train (-1), test (-1); repo https://github.com/stellaxu/DHAN

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Measures the probability that a randomly chosen positive instance (reviewed item) is ranked higher than a randomly chosen negative instance (unreviewed item).
• RelaImpr — range: percent
  • Relative Improvement over a baseline model: ((AUC(model) - 0.5) / (AUC(base) - 0.5) - 1) * 100%. Used to quantify performance gain relative to a reference model.

Input / output format

Input: Sequence of the first k historically reviewed items (positive samples) and a candidate item (positive or negative sample) for each user.

Output: A predicted click-through rate (CTR) probability or relevance score for the candidate item.

Scoring recipe

def compute_auc(y_true, y_pred):
    return roc_auc_score(y_true, y_pred)

def compute_rela_impr(auc_model, auc_base):
    return ((auc_model - 0.5) / (auc_base - 0.5) - 1) * 100

# y_true: 1 for reviewed, 0 for unreviewed
# y_pred: model's predicted probability
auc = compute_auc(y_true, y_pred)
rela_impr = compute_rela_impr(auc, auc_baseline)

Common pitfalls

• AUC is computed on sampled negative items, so results depend heavily on the negative sampling strategy (items not reviewed by the user).
• RelaImpr measures relative gain over a specific baseline (DIN), not absolute performance; comparing across papers requires matching baselines.
• The task uses a sequential prediction setup (predict k+1 from k), so evaluation must respect temporal order to avoid data leakage.

Evidence (verbatim from paper)

Our task is to predict $(\mathrm{k} + 1)$ -th reviewed item by using the first $k$ reviewed items. For each user, we construct positive sample by taking user's historical reviews, and generate a negative sample from items not reviewed by this user. RelaImpr[3] metric is used to measure the relative improvement over models, which is defined as: $$ \text{RelaImpr} = \left(\frac{\text{AUC(measured model)} - 0.5}{\text{AUC(base model)} - 0.5} -1\right)\times 100%. $$

Citation

@misc{xu2020dhan,
  title={Deep Interest with Hierarchical Attention Network for Click-Through Rate Prediction},
  author={Weinan Xu et al. (2020)},
  year={2020},
  note={arXiv:2005.12981}
}

• arXiv: 2005.12981