Cross Domain Ctr Eval

cross-domain-ctr-eval

Enhancing Cross-domain Click-Through Rate Prediction via Explicit Feature Augmentation — Xu Chen et al. (2023) (arXiv:2312.00078, 2023)

What this evaluates

Evaluates cross-domain knowledge transfer for click-through rate (CTR) prediction by measuring how well a model trained on a source domain generalizes to a target domain with non-overlapping features. It probes context-aware feature translation and explicit knowledge augmentation in recommendation systems.

Datasets

• Amazon — total ?; splits: train (-1), val (-1), test (-1); repo https://jmcauley.ucsd.edu/data/amazon/
• Taobao — total ?; splits: train (-1), val (-1), test (-1); repo https://tianchi.aliyun.com/dataset/56333
• Alibaba Production — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the probability that a randomly chosen positive instance is ranked higher than a negative one.
• CTR — range: percent
  • Click-Through Rate: ratio of total clicks to total impressions, reported as a percentage.
• GMV — range: other
  • Gross Merchandise Value: sum of transaction values for all completed purchases.

Input / output format

Input: User and item IDs embedded as 64-dim vectors; discrete contextual features (e.g., age, gender, category, brand) encoded as one-hot or multi-hot vectors; item names processed via GloVe-6B word embeddings.

Output: Binary click probability or logit score for ranking; final prediction is binary (click/no-click or positive/negative feedback).

Scoring recipe

def compute_metrics(y_true, y_pred, impressions, prices):
    auc = roc_auc_score(y_true, y_pred)
    ctr = (y_true.sum() / impressions) * 100
    gmv = (y_true * prices).sum()
    return {'AUC': auc, 'CTR': ctr, 'GMV': gmv}

Common pitfalls

• Data must be split chronologically (8:1:1 train/val/test), not randomly, to prevent data leakage from future user behaviors.
• Cross-domain baselines like KEEP require a 'super domain' covering most users/items; public benchmarks lack this, so KEEP is only evaluated on production data.
• Non-overlapping feature fields across domains mean shared embedding layers cannot be used; models must rely on explicit feature translation or independent feature extractors.

Evidence (verbatim from paper)

In our experiments on public benchmarks, we split the data into train, validation, and test sets with the common 8:1:1 setting according to chronological order. The experiments are conducted multiple times and the mean value is taken as the model performance. ... The online evaluation metrics are real CTR, deal number and GMV. ... AUC indicates the offline evaluation metric.

Citation

@misc{chen2023enhancing,
  title={Enhancing Cross-domain Click-Through Rate Prediction via Explicit Feature Augmentation},
  author={Xu Chen et al. (2023)},
  year={2023},
  note={arXiv:2312.00078}
}

• arXiv: 2312.00078