Cross Domain Sequential Rec Eval

cross-domain-sequential-rec-eval

Mixed Attention Network for Cross-domain Sequential Recommendation — Guanyu Lin et al. (arXiv:2311.08272, 2023)

What this evaluates

This evaluation probes a model's ability to perform cross-domain sequential recommendation by leveraging user interaction histories across two domains, even when user overlap is minimal or absent. It measures how well the model captures domain-specific and shared sequential patterns to rank candidate items accurately.

Datasets

• Micro Video — total ?; splits: (unstated)
• Amazon — total ?; splits: (unstated)

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance.
• MRR — range: [0, 1]
  • Mean Reciprocal Rank. The average of the reciprocal of the rank of the first relevant item in the predicted list.
• NDCG — range: [0, 1]
  • Normalized Discounted Cumulative Gain. Measures ranking quality by discounting the relevance of items based on their position in the list, normalized by the ideal DCG.
• WAUC — range: [0, 1]
  • Weighted AUC. AUC metric weighted by user activity or sequence length to account for data sparsity and user engagement differences.

Input / output format

Input: User item interaction sequences (history) from one or two domains, optionally with the second domain's sequence empty for single-domain baselines.

Output: Predicted relevance scores or ranked list of candidate items for the next interaction.

Scoring recipe

def compute_metrics(predictions, gold):
    auc_scores = [roc_auc_score(g, p) for g, p in zip(gold, predictions)]
    mrr_scores = [reciprocal_rank(g, p) for g, p in zip(gold, predictions)]
    ndcg_scores = [ndcg_at_k(g, p, k=10) for g, p in zip(gold, predictions)]
    return {
        'AUC': np.mean(auc_scores),
        'MRR': np.mean(mrr_scores),
        'NDCG@10': np.mean(ndcg_scores)
    }

Common pitfalls

• Assuming cross-domain recommenders require fully overlapped users; the paper shows baselines like PiNet and DASL degrade significantly without user overlap.
• Training both domains simultaneously on a shared backbone causes optimization conflict and negative transfer, which the authors explicitly avoid by using separate local and global modules.

Evidence (verbatim from paper)

All models are evaluated on two popular accuracy metrics AUC and GAUC [8], as well as two ranking metrics, MRR and NDCG [2].

Citation

@misc{lin2023mixed,
  title={Mixed Attention Network for Cross-domain Sequential Recommendation},
  author={Guanyu Lin et al.},
  year={2023},
  note={arXiv:2311.08272}
}

• arXiv: 2311.08272