Sequential Rec Temporal Eval

sequential-rec-temporal-eval

Sequential Recommendation on Temporal Proximities with Contrastive Learning and Self-Attention — Jung et al. (2024) (arXiv:2402.09784, 2024)

What this evaluates

Evaluates a model's ability to predict the next item in a user's sequential interaction history by leveraging both temporal proximity across users and within-user sequence dynamics.

Datasets

• Amazon (Beauty, Book, Video) — total ?; splits: train (-1), val (-1), test (-1); repo http://jmcauley.ucsd.edu/data/amazon/
• Steam — total ?; splits: train (-1), val (-1), test (-1); repo https://cseweb.ucsd.edu/~jmcauley/datasets.html#steam_data

Metrics

• NDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Computes the discounted gain of the ground-truth item at its predicted rank, capped at 10. Formula: 1/log2(rank+1).
• HR@10 — range: [0, 1]
  • Hit Ratio at rank 10. Binary indicator that is 1 if the ground-truth item appears in the top-10 recommended items, else 0.

Input / output format

Input: Chronologically sorted sequence of user-item interactions with timestamps.

Output: Ranked list of 10 recommended items.

Scoring recipe

def score(recommended_list, gold_item):
    hr = 1.0 if gold_item in recommended_list[:10] else 0.0
    rank = recommended_list.index(gold_item) + 1 if gold_item in recommended_list else 11
    ndcg = 1.0 / math.log2(rank) if gold_item in recommended_list else 0.0
    return {'HR@10': hr, 'NDCG@10': ndcg}

Common pitfalls

• Negative sampling strictly uses 100 randomly selected non-interacted items per user for ranking.
• Sequential split assigns the last item to test and the second-to-last to validation, not random splits.
• Dataset filtering thresholds vary by domain (e.g., Book requires >=30 user interactions, Steam >=10), affecting train/val/test sizes.

Evidence (verbatim from paper)

For each user sequence, the last item in the sequence was used for the test, while the item just before the last one for the validation. To ensure a fair and simple evaluation, we adopted the negative sampling strategy in (Sun et al., 2019). For each user u, we randomly select about 100 items they haven’t interacted with and rank them alongside the ground-truth item. We used two measures widely used for the evaluation of ranked item lists: Hit Ratio (HR@K) and Normalized Discounted Cumulative Gain (NDCG@K). We set K to 10, meaning that the model recommends 10 items for each user.

Citation

@misc{jung2024temproxrec,
  title={Sequential Recommendation on Temporal Proximities with Contrastive Learning and Self-Attention},
  author={Jung et al. (2024)},
  year={2024},
  note={arXiv:2402.09784}
}

• arXiv: 2402.09784