Sequential Recsys Eval

sequential-recsys-eval

A Reproducible Analysis of Sequential Recommender Systems — Betello et al. (2024) (arXiv:2408.03873, 2024)

What this evaluates

Evaluates the performance and reproducibility of sequential recommender system (SRS) models across multiple user-item interaction datasets. It probes how architectural choices, hyperparameter settings, and training configurations affect ranking metrics and computational emissions.

Datasets

• Beauty — total 7113; splits: train (-1), val (-1), test (-1); repo https://github.com/antoniopurificato/recsys_repro_conf
• FS-NYC — total 179468; splits: train (-1), val (-1), test (-1); repo https://github.com/antoniopurificato/recsys_repro_conf
• FS-TKY — total 494807; splits: train (-1), val (-1), test (-1); repo https://github.com/antoniopurificato/recsys_repro_conf
• ML-100k — total 99287; splits: train (-1), val (-1), test (-1); repo https://github.com/antoniopurificato/recsys_repro_conf
• ML-1M — total 999611; splits: train (-1), val (-1), test (-1); repo https://github.com/antoniopurificato/recsys_repro_conf
• ML-20M — total 19984024; splits: train (-1), val (-1), test (-1); repo https://github.com/antoniopurificato/recsys_repro_conf

Metrics

• NDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Calculated as DCG@10 divided by the ideal DCG@10, where DCG discounts relevance by the logarithm of the rank position.
• Recall@20 — range: [0, 1]
  • Proportion of relevant items found in the top-20 recommended items out of all relevant items in the ground truth.
• Precision — range: [0, 1]
  • Proportion of recommended items in the top-K list that are relevant.
• MAP — range: [0, 1]
  • Mean Average Precision across all users, averaging the precision at each relevant item's rank position.

Input / output format

Input: User-item interaction sequences (lists of item IDs) representing a user's historical behavior, processed to generate predictions for the next item(s).

Output: Ranked list of candidate items (or top-K recommendations) for the next interaction, evaluated against ground-truth next items.

Scoring recipe

def compute_metrics(predictions, ground_truth, k=10):
    # predictions: list of item IDs ranked by model
    # ground_truth: list of relevant item IDs (usually 1 for next-item rec)
    # Recall@k
    recall = len(set(predictions[:k]) & set(ground_truth)) / max(len(ground_truth), 1)
    # NDCG@k
    dcg = sum(1.0 / math.log2(i + 2) for i, item in enumerate(predictions[:k]) if item in ground_truth)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(ground_truth), k)))
    ndcg = dcg / idcg if idcg > 0 else 0.0
    return ndcg, recall

Common pitfalls

• Using different loss functions (e.g., BPR vs BCE) or training durations across models, which heavily skews performance rankings.
• Inconsistent data splitting (e.g., random vs temporal last-interaction split) or negative sampling strategies.
• Ignoring computational emissions and training time, which are critical for fair comparison in modern SRS evaluations.

Evidence (verbatim from paper)

To evaluate the performance of sequential recommendation algorithms, we use four widely used metric, also common in Information Retrieval (IR): Precision, Recall, NDCG and MAP. ... For testing, as in (Sun et al., [2019]; Kang and McAuley, [2018]), we keep the last interaction for each user, while for the validation set, the second to last action is retained. All remaining interactions contribute to the training set.

Citation

@misc{betello2024reproducible,
  title={A Reproducible Analysis of Sequential Recommender Systems},
  author={Betello et al. (2024)},
  year={2024},
  note={arXiv:2408.03873}
}

• arXiv: 2408.03873