Session Rec Rnn Eval

session-rec-rnn-eval

Session-based Recommendations with Recurrent Neural Networks — Hidasi et al. (2015) (arXiv:1511.06939, 2015)

What this evaluates

Evaluates a model's ability to predict the next item in a user session based on sequential click or watch history. It probes the capability to capture short-term sequential dependencies and maintain context within session-based recommendation scenarios.

Datasets

• RSC15 — total ?; splits: train (7966257), test (15324)
• VIDEO — total ?; splits: train (3000000), test (37000)

Metrics

• recall@20 (primary) — range: [0, 1]
  • Proportion of test cases where the ground-truth next item appears in the model's top-20 ranked recommendations. It does not penalize based on the exact rank within the top-20.
• MRR@20 — range: [0, 1]
  • Average of reciprocal ranks of the ground-truth items across test cases. The reciprocal rank is set to zero if the item's rank exceeds 20.

Input / output format

Input: A sequence of item IDs representing a user's session history, processed one event at a time.

Output: A ranked list of candidate item IDs in descending order of predicted relevance score.

Scoring recipe

def compute_metrics(predictions, gold_items):
    recall_hits = 0
    mrr_sum = 0.0
    for pred_list, gold in zip(predictions, gold_items):
        if gold in pred_list[:20]:
            recall_hits += 1
        rank = pred_list.index(gold) + 1 if gold in pred_list else 21
        mrr_sum += (1.0 / rank) if rank <= 20 else 0.0
    return recall_hits / len(gold_items), mrr_sum / len(gold_items)

Common pitfalls

• Test set clicks are filtered out if the clicked item was not present in the training set, which can artificially reduce test size for collaborative filtering baselines.
• Sessions of length 1 are removed from both train and test sets, potentially biasing evaluation against models that struggle with single-item contexts.
• For the VIDEO dataset, evaluation is restricted to ranking the target item against only the 30,000 most popular items rather than the full 330,000-item catalog.

Evidence (verbatim from paper)

Therefore, our primary evaluation metric is recall@20 that is the proportion of cases having the desired item amongst the top-20 items in all test cases. Recall does not consider the actual rank of the item as long as it is amongst the top-N. ... The second metric used in the experiments is MRR@20 (Mean Reciprocal Rank). That is the average of reciprocal ranks of the desired items. The reciprocal rank is set to zero if the rank is above 20.

Citation

@misc{hidasi2015session,
  title={Session-based Recommendations with Recurrent Neural Networks},
  author={Hidasi et al. (2015)},
  year={2015},
  note={arXiv:1511.06939}
}

• arXiv: 1511.06939