Recsys Challenge 2015 Eval

recsys-challenge-2015-eval

Recurrent Latent Variable Networks for Session-Based Recommendation — Chatzis et al. (2017) (arXiv:1706.04026, 2017)

What this evaluates

Evaluates session-based recommendation models by predicting the next item in a user's clickstream sequence. It probes the model's ability to capture temporal dynamics and handle data sparsity in e-commerce sessions.

Datasets

• RecSys Challenge 2015 — total ?; splits: train (7966257), test (5324)

Metrics

• Recall@20 (primary) — range: [0, 1]
  • Measures the fraction of test sessions where the ground-truth next item appears in the model's top-20 recommended list.
• MRR@20 — range: [0, 1]
  • Computes the average reciprocal rank of the ground-truth item within the top-20 recommendations, assigning a score of 0 if the item is not in the top-20.

Input / output format

Input: A sequence of item IDs representing a user's clickstream session.

Output: A ranked list of candidate item IDs (top-20 for evaluation).

Scoring recipe

def compute_metrics(predictions, ground_truth):
    recalls = []
    mrrs = []
    for pred, gt in zip(predictions, ground_truth):
        recalls.append(1.0 if gt in pred[:20] else 0.0)
        rank = None
        for i, item in enumerate(pred[:20]):
            if item == gt:
                rank = i + 1
                break
        mrrs.append(1.0 / rank if rank else 0.0)
    return sum(recalls) / len(recalls), sum(mrrs) / len(mrrs)

Common pitfalls

• The official test set from RecSys Challenge 2015 lacks ground-truth labels, so the authors split the original training data to create a test set, which may not reflect the original challenge's intended evaluation protocol.
• Comparisons with methods like M2/M4 are indirect because those baselines used data augmentation and pretraining, which the authors explicitly avoided to keep the comparison fair but limits direct performance parity expectations.

Evidence (verbatim from paper)

To quantitatively assess the performance of our approach, we employ two commonly used evaluation metrics, namely Recall@20 and Mean Reciprocal Rank (MRR)@20. The former metric expresses the frequency at which the desired (groundtruth) item in the test data makes it to the 20 highest ranked items suggested by the evaluated approach. Hence, this metric allows for modeling and assessing certain practical scenarios where there is no highlighting of recommendations; what matters is the desired item being included in a short list of recommendations, rather than the absolute order that these items are presented to the user. On the other hand, MRR@20 describes the average predicted score of the desired items in the test data, with the score values set to zero if the desired item does not make it to the top-20 list of ranked items.

Citation

@misc{chatzis2017recurrent,
  title={Recurrent Latent Variable Networks for Session-Based Recommendation},
  author={Chatzis et al. (2017)},
  year={2017},
  note={arXiv:1706.04026}
}

• arXiv: 1706.04026