Contrastvae Eval

contrastvae-eval

ContrastVAE: Contrastive Variational AutoEncoder for Sequential Recommendation — Wang et al. (2022) (arXiv:2209.00456, 2022)

What this evaluates

Evaluates sequential recommendation models on Amazon review datasets, measuring ranking quality for next-item prediction. It specifically probes performance on long-tail items, sequence sparsity, and robustness to noisy inputs.

Datasets

• Amazon Beauty — total 198502; splits: train (-1), val (-1), test (-1)
• Amazon Toys — total 167597; splits: train (-1), val (-1), test (-1)
• Amazon Tools — total 134476; splits: train (-1), val (-1), test (-1)
• Amazon Office — total 53258; splits: train (-1), val (-1), test (-1)

Metrics

• Recall@20 (primary) — range: [0, 1]
  • Fraction of relevant items found in the top-20 recommended items. Calculated as |predicted_top20 ∩ relevant| / |relevant|.
• Recall@40 — range: [0, 1]
  • Fraction of relevant items found in the top-40 recommended items. Calculated as |predicted_top40 ∩ relevant| / |relevant|.
• NDCG@20 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 20. Measures ranking quality by discounting gains logarithmically based on position.
• NDCG@40 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 40. Measures ranking quality by discounting gains logarithmically based on position.

Input / output format

Input: A chronological sequence of user-item interactions (implicit feedback) sorted by timestamp.

Output: A ranked list of top-N candidate items for recommendation.

Scoring recipe

def recall_at_n(pred_list, relevant_set, n):
    top_n = set(pred_list[:n])
    return len(top_n & relevant_set) / len(relevant_set)

def ndcg_at_n(pred_list, relevant_set, n):
    dcg = sum(1.0 / math.log2(i + 2) for i, item in enumerate(pred_list[:n]) if item in relevant_set)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(relevant_set), n)))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Failing to filter users and items with fewer than 5 interactions before splitting the data.
• Using random splits instead of chronological order (last item for test, penultimate for validation, rest for training).
• Treating rating scores as explicit feedback instead of implicit binary interactions.

Evidence (verbatim from paper)

We compute each user's relevance scores for all items and choose items with top-N scores for the recommendation. Then we adopt two widely used top-N metrics, Recall and NDCG, as our top-N ranking evaluation metrics. We report the experimental results when N = 20 and N = 40.

Citation

@misc{wang2022contrastvae,
  title={ContrastVAE: Contrastive Variational AutoEncoder for Sequential Recommendation},
  author={Wang et al. (2022)},
  year={2022},
  note={arXiv:2209.00456}
}

• arXiv: 2209.00456