Recsys2015 Session Recomm Eval

recsys2015-session-recomm-eval

Improved Recurrent Neural Networks for Session-based Recommendations — Tan et al. (2016) (arXiv:1606.08117, 2016)

What this evaluates

Evaluates session-based recommendation models by predicting the next item in a user's browsing sequence. It measures ranking quality and prediction efficiency to assess accuracy and deployability in real-time recommender systems.

Datasets

• RecSys Challenge 2015 dataset — total 7981491; splits: train (7966257), test (15234)

Metrics

• Recall@20 (primary) — range: [0, 1]
  • The fraction of test sessions where the true next item appears in the model's top-20 predicted items. Calculated as the average of binary indicators (1 if in top-20, 0 otherwise) across all test sessions.
• MRR@20 — range: [0, 1]
  • Mean Reciprocal Rank at 20. For each session, if the true next item is in the top-20 list, its reciprocal rank (1/rank) is recorded; otherwise 0. The metric is the average across all test sessions.

Input / output format

Input: A sequence of items in a user session, fed item-by-item to the RNN model.

Output: A ranked list of the top-20 candidate items for the next position in the session. For M1-M3, derived from softmax probabilities; for M4, derived from cosine similarity to item embeddings.

Scoring recipe

def compute_metrics(predictions, gold_item):
    if gold_item in predictions[:20]:
        rank = predictions.index(gold_item) + 1
        recall = 1.0
        mrr = 1.0 / rank
    else:
        recall = 0.0
        mrr = 0.0
    return recall, mrr

# Aggregate over all test sessions
total_recall = sum(compute_metrics(pred, gold)[0] for pred, gold in test_data)
total_mrr = sum(compute_metrics(pred, gold)[1] for pred, gold in test_data)
avg_recall = total_recall / len(test_data)
avg_mrr = total_mrr / len(test_data)

Common pitfalls

• M4 generates the top-20 list using cosine similarity to item embeddings rather than softmax probabilities, which changes the ranking procedure compared to M1-M3.
• The dataset uses a strict chronological split (last day = test) to prevent data leakage, so random shuffling or standard train/val/test splits will invalidate the evaluation.
• Sequences are truncated or padded to exactly 19 time-steps during training, which may artificially limit performance on sessions longer than 19 items.

Evidence (verbatim from paper)

The evaluation metrics used were Recall@20 and Mean Reciprocal Rank (MRR)@20. These metrics are designed for the recommendation setting, as we usually want to make multiple recommendations for each user. For M1-M3, we take the top 20 most probable items directly from the softmax outputs. For M4, we compute the cosine distance of the model output against the embedding of items, and take the top 20 closest items.

Citation

@misc{tan2016improved,
  title={Improved Recurrent Neural Networks for Session-based Recommendations},
  author={Tan et al. (2016)},
  year={2016},
  note={arXiv:1606.08117}
}

• arXiv: 1606.08117