Seq Rec Aug Eval

seq-rec-aug-eval

Is Contrastive Learning Necessary? A Study of Data Augmentation vs Contrastive Learning in Sequential Recommendation — Zhou et al. (2024) (arXiv:2403.11136, 2024)

What this evaluates

This evaluation protocol benchmarks sequential recommendation models by comparing sequence-level data augmentation strategies against contrastive learning baselines. It probes a model's ability to capture user intent from interaction sequences and generate accurate item rankings under varying data sparsity, sequence lengths, and cold-start conditions.

Datasets

• Amazon Beauty — total ?; splits: train (-1), val (-1), test (-1)
• Amazon Sports — total ?; splits: train (-1), val (-1), test (-1)
• Yelp — total ?; splits: train (-1), val (-1), test (-1)
• ML-1m — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Recall@K / NDCG@K (K∈{10, 20}) (primary) — range: percent
  • Recall@K measures the fraction of the single test item appearing in the top-K predicted list. NDCG@K evaluates ranking quality by weighting the test item by the inverse logarithm of its predicted rank. Both metrics are averaged across all users and reported as percentages.

Input / output format

Input: An ordered sequence of user interaction item IDs, truncated or padded to a maximum length (50 for Beauty/Sports/Yelp, 200 for ML-1m).

Output: A score or rank for every item in the complete catalog, from which the top-K items are extracted for evaluation.

Scoring recipe

def compute_metrics(pred_scores, test_item, K):
    # Get top-K item indices from full catalog scores
    top_k = get_top_k_indices(pred_scores, K)
    # Recall@K: 1 if test item is in top-K, else 0
    recall = 1.0 if test_item in top_k else 0.0
    # NDCG@K: only one test item per user in leave-one-out
    dcg = 0.0
    for i, item in enumerate(top_k):
        if item == test_item:
            dcg += 1.0 / math.log2(i + 2)
    idcg = 1.0 / math.log2(2)
    ndcg = dcg / idcg
    return recall, ndcg

Common pitfalls

• Calculating ranking metrics over a sampled subset of negative items instead of the complete item set, which artificially inflates performance and violates the paper's protocol.
• Failing to apply the strict leave-one-out split where exactly the last two items are reserved for validation and testing respectively.
• Ignoring dataset-specific sequence length limits when configuring augmentation window sizes, leading to inconsistent experimental conditions across datasets.

Evidence (verbatim from paper)

In our experiment, we select Recall@K and NDCG@K as the evaluation metrics for different methods, where K can be either 10 or 20. These two metrics are widely adopted in existing sequential recommendation research [4, 27, 37, 50]. Regarding dataset partitioning, we utilize a leave-one-out approach: the final two items within each user interaction sequence are allocated to the validation and test sets, respectively, while the remainder of the items is utilized for model training. To ensure fair comparison, we follow the suggestion of [5, 20] to calculate the ranking results over the complete item set rather than a sampled subset.

Citation

@misc{zhou2024contrastive,
  title={Is Contrastive Learning Necessary? A Study of Data Augmentation vs Contrastive Learning in Sequential Recommendation},
  author={Zhou et al. (2024)},
  year={2024},
  note={arXiv:2403.11136}
}

• arXiv: 2403.11136