Dsin Ctr Eval

dsin-ctr-eval

Deep Session Interest Network for Click-Through Rate Prediction — Feng et al. (2019) (arXiv:1905.06482, 2019)

What this evaluates

This benchmark evaluates a model's ability to predict click-through rates (CTR) by leveraging session-aware user behavior sequences. It probes how well a system can decompose historical interactions into time-separated sessions, model cross-session interest evolution, and adaptively weight session interests relative to a target item.

Datasets

• Advertising Dataset — total 26000000; splits: train (-1), test (-1)
• Recommender Dataset — total 6000000000; splits: train (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • AUC = (1 / (m^+ * m^-)) * Σ_{x^+ ∈ D^+} Σ_{x^- ∈ D^-} I(f(x^+) > f(x^-)), where D^+ and D^- are positive and negative examples, f(·) is the model's prediction score, and I(·) is the indicator function. It measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance.

Input / output format

Input: User ID, target item ID, and a sequence of up to 200 historical user behaviors (display/click logs) partitioned into time-separated sessions.

Output: A continuous prediction score (probability) representing the likelihood of the user clicking the target item.

Scoring recipe

def compute_auc(predictions, labels):
    pos_scores = [p for p, l in zip(predictions, labels) if l == 1]
    neg_scores = [p for p, l in zip(predictions, labels) if l == 0]
    if not pos_scores or not neg_scores:
        return 0.0
    concordant = sum(1 for ps in pos_scores for ns in neg_scores if ps > ns)
    return concordant / (len(pos_scores) * len(neg_scores))

Common pitfalls

• The dataset splits are strictly time-based (specific dates in 2017 and 2018), not random, so temporal leakage must be avoided.
• User behavior sequences are capped at the recent 200 interactions; truncating or padding incorrectly will alter session boundaries and model performance.
• AUC is computed over all positive/negative pairs in the test set; using approximation methods or incorrect negative sampling can skew results.

Evidence (verbatim from paper)

AUC (Area Under ROC Curve) reflects the ranking ability of the model. It is defined as follows: $$ \mathrm {A U C} = \frac {1}{m ^ {+} m ^ {-}} \sum_ {x ^ {+} \in D ^ {+}} \sum_ {x ^ {-} \in D ^ {-}} (I (f (x ^ {+}) > f (x ^ {-}))) \tag {11} $$ where $D^{+}$ is the collection of all positive examples, $D^{-}$ is the collection of all negative examples, $f(\cdot)$ is the result of the model's prediction of the sample $\mathbf{x}$ and $I(\cdot)$ is the indicator function.

Citation

@misc{feng2019dsin,
  title={Deep Session Interest Network for Click-Through Rate Prediction},
  author={Feng et al. (2019)},
  year={2019},
  note={arXiv:1905.06482}
}

• arXiv: 1905.06482