Diffurec Eval

diffurec-eval

DiffuRec: A Diffusion Model for Sequential Recommendation — Li et al. (2023) (arXiv:2304.00686, 2023)

What this evaluates

Evaluates a model's ability to predict the next item in a user's interaction sequence by capturing dynamic preferences and multi-aspect item representations. It probes sequential recommendation performance under varying sequence lengths and item popularities.

Datasets

• Amazon Beauty — total 22363; splits: train (-1), val (-1), test (-1)
• Amazon Toys — total 19412; splits: train (-1), val (-1), test (-1)
• Movielens-1M — total 6040; splits: train (-1), val (-1), test (-1)
• Steam — total 281428; splits: train (-1), val (-1), test (-1)

Metrics

• HR@K (primary) — range: [0, 1]
  • Hit Rate at K. Returns 1 if the target item appears in the top-K ranked list, else 0. Averaged across all test instances.
• NDCG@K — range: [0, 1]
  • Normalized Discounted Cumulative Gain at K. Returns 1/log2(rank+1) if the target item is in the top-K list, else 0. Averaged across all test instances.

Input / output format

Input: Chronologically ordered sequence of user-item interactions (items i_1 to i_{n-2}) used to predict the next item i_n.

Output: A ranked list of all candidate items, from which the top-K predictions are extracted.

Scoring recipe

def compute_metrics(ranked_items, target_item, K):
    hit = 1.0 if target_item in ranked_items[:K] else 0.0
    ndcg = 0.0
    if hit:
        rank = ranked_items.index(target_item) + 1
        if rank <= K:
            ndcg = 1.0 / math.log2(rank + 1)
    return hit, ndcg

Common pitfalls

• Using negative sampling instead of ranking all candidate items, which the paper explicitly notes causes inconsistency when the number of negative items is small.
• Ignoring the chronological leave-one-out split protocol, which is critical for temporal sequential recommendation evaluation.
• Not applying the dataset-specific maximum sequence length limits (50 for Beauty/Toys/Steam, 200 for MovieLens-1M) during preprocessing.

Evidence (verbatim from paper)

Following the common data preprocessing method [25, 42, 48], we treat all reviews or ratings as implicit feedback (i.e., a user-item interaction) and chronologically organize them by their timestamps. ... we adopt a leave-one-out strategy for performance evaluation. To be specific, for all the datasets, given a sequence S = {i1, i2, ..., in}, we utilize the most recent interaction (in) for testing, the penultimate interaction (in-1) for model validation, and the earlier ones {(i1, i2, ..., in-2)} for model training. ... As for the evaluation metric, we evaluate all models with HR@K (Hit Rate) and NDCG@K (Normalized Discounted Cumulative Gain). We report the experimental results with K = {5, 10, 20}. HR@K represents the proportion of the hits recommended among the top-K list. NDCG@K further evaluates the ranking performance by considering the ranking positions of these hits. The NDCG@K is set to 0 when the rank exceeds K. Because the evaluation with sampling may cause inconsistency when the number of negative items is small [26], we rank all candidate items for target item prediction.

Citation

@misc{li2023diffurec,
  title={DiffuRec: A Diffusion Model for Sequential Recommendation},
  author={Li et al. (2023)},
  year={2023},
  note={arXiv:2304.00686}
}

• arXiv: 2304.00686