Sasrec Sequential Rec Eval

sasrec-sequential-rec-eval

Self-Attentive Sequential Recommendation — Wang-Cheng Kang et al. (2018) (arXiv:1808.09781, 2018)

What this evaluates

Evaluates a model's ability to predict the next item in a user's interaction sequence based on historical behavior. It probes the model's capacity to capture long-range dependencies and adapt to varying data sparsity across different domains.

Datasets

• Amazon (Beauty) — total ?; splits: train (-1), val (-1), test (-1)
• Amazon (Games) — total ?; splits: train (-1), val (-1), test (-1)
• Steam — total 7793069; splits: train (-1), val (-1), test (-1)
• MovieLens-1M — total 1000000; splits: train (-1), val (-1), test (-1)

Metrics

• Recall@K (primary) — range: [0, 1]
  • Standard sequential recommendation metric; fraction of users for whom the ground-truth next item appears in the top-K predicted items.

Input / output format

Input: A sequence of user-item interactions (reviews/ratings) ordered by timestamp, including all training actions and the validation action.

Output: A single predicted item (the next interaction in the sequence).

Scoring recipe

def evaluate(predictions, ground_truth, K):
    hits = 0
    for pred_list, true_item in zip(predictions, ground_truth):
        if true_item in pred_list[:K]:
            hits += 1
    return hits / len(ground_truth)

Common pitfalls

• Users and items with fewer than 5 related actions are discarded, significantly altering dataset statistics.
• During testing, the input sequence explicitly includes the validation action alongside training actions, not just the training history.
• Splits are chronological per-user (leave-one-out), not random, so temporal leakage must be avoided.

Evidence (verbatim from paper)

We followed the same preprocessing procedure from [1], [19], [21]. For all datasets, we treat the presence of a review or rating as implicit feedback (i.e., the user interacted with the item) and use timestamps to determine the sequence order of actions. We discard users and items with fewer than 5 related actions. For partitioning, we split the historical sequence $S^u$ for each user $u$ into three parts: (1) the most recent action $S_{|S^u|}^u$ for testing, (2) the second most recent action $S_{|S^u| - 1}^u$ for validation, (3) all remaining actions for training. Note that during testing, the input sequences contain training actions and the validation action.

Citation

@misc{kang2018sasrec,
  title={Self-Attentive Sequential Recommendation},
  author={Wang-Cheng Kang et al. (2018)},
  year={2018},
  note={arXiv:1808.09781}
}

• arXiv: 1808.09781