Samrec Eval

samrec-eval

Enhancing Transformers without Self-supervised Learning: A Loss Landscape Perspective in Sequential Recommendation — Lai et al. (2023) (arXiv:2308.10347, 2023)

What this evaluates

Evaluates sequential recommendation models on their ability to predict the next item in a user's chronological interaction history. It specifically probes whether sharpness-aware minimization improves generalization and data efficiency compared to standard Transformers and self-supervised baselines.

Datasets

• Amazon-Beauty — total ?; splits: train (-1), val (-1), test (-1)
• Amazon-Sports — total ?; splits: train (-1), val (-1), test (-1)
• Amazon-Toys — total ?; splits: train (-1), val (-1), test (-1)
• Yelp — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• HR@10 (primary) — range: [0, 1]
  • 1 if the ground-truth item appears in the top-10 predicted items, else 0.
• NDCG@10 — range: [0, 1]
  • Sum of 1/log2(rank+1) for each hit in the top-10 list, normalized by the ideal DCG@10 (which equals 1 for a single positive item).

Input / output format

Input: A chronological sequence of user-item interaction IDs.

Output: A ranked list of top-10 candidate item IDs.

Scoring recipe

def evaluate(predictions, ground_truth):
    hr = 1.0 if ground_truth in predictions else 0.0
    if ground_truth in predictions:
        rank = predictions.index(ground_truth) + 1
        ndcg = 1.0 / math.log2(rank + 1)
    else:
        ndcg = 0.0
    return hr, ndcg

Common pitfalls

• Failing to apply the 5-core filtering before splitting the data.
• Not sorting user interactions by timestamp before creating train/val/test splits.
• Reporting results from a single run instead of averaging over 10 independent runs.

Evidence (verbatim from paper)

We use two widely used metrics, Hit Ratio (HR) and Normalized Discounted Cumulative Gain (NDCG), to evaluate performance. We repeat the experiments 10 times independently and report the average results for HR@10 and NDCG@10.

Citation

@misc{lai2023enhancing,
  title={Enhancing Transformers without Self-supervised Learning: A Loss Landscape Perspective in Sequential Recommendation},
  author={Lai et al. (2023)},
  year={2023},
  note={arXiv:2308.10347}
}

• arXiv: 2308.10347