Session Rec Hrnn Eval

session-rec-hrnn-eval

Personalizing Session-based Recommendations with Hierarchical Recurrent Neural Networks — Quadrana et al. (2017) (arXiv:1706.04148, 2017)

What this evaluates

This evaluation probes a model's ability to perform personalized session-based sequential recommendation by predicting the next item a user will interact with. It measures how well the model leverages both intra-session behavior and cross-session user history to rank relevant items in a top-5 list.

Datasets

• XING — total 546862; splits: train (488576), test (58286)
• VIDEO — total 825449; splits: train (745482), test (79967)

Metrics

• Recall@5 — range: [0, 1]
  • Proportion of test cases in which the relevant (gold) item appears in the top-5 recommended items.
• Precision@5 — range: [0, 1]
  • Fraction of correct recommendations in the top-5 positions of each recommendation list.
• MRR@5 (primary) — range: [0, 1]
  • Reciprocal rank of the relevant item, where the reciprocal rank is manually set to zero if the rank is greater than 5.

Input / output format

Input: A sequence of item IDs representing the current user session, optionally preceded by a sequence of all prior sessions from the same user's history for bootstrapping personalized models.

Output: A ranked list of candidate items (top-5 recommendations) for the next interaction.

Scoring recipe

def compute_metrics(predictions, gold_item, k=5):
    top_k = predictions[:k]
    hit = 1 if gold_item in top_k else 0
    precision = hit / k
    rank = predictions.index(gold_item) + 1 if gold_item in predictions else k + 1
    mrr = 1.0 / rank if rank <= k else 0.0
    return hit, precision, mrr

Common pitfalls

• Forgetting to bootstrap personalized models (HRNN/RNN Concat) with all preceding user sessions before evaluation, which is required to properly initialize internal representations.
• Discarding the first prediction in RNN Concat test sessions, as it is the only baseline capable of recommending the first event in a session.
• Evaluating on raw data without applying the specified filtering thresholds (items with support <20/10, sessions <3 interactions, users <5 sessions).

Evidence (verbatim from paper)

We therefore evaluate the recommendation quality in terms of Recall@5, Precision@5 and Mean Reciprocal Rank (MRR@5). In sequential next-item prediction, Recall@5 is equivalent to the hit-rate metric, and it measures the proportion of cases out of all test cases in which the relevant item is amongst the top-5 items. This is an accurate model for certain practical scenarios where no recommendation is highlighted and their absolute order does not matter, and strongly correlates with important KPIs such as CTR. Precision@5 measures the fraction of correct recommendations in the top-5 positions of each recommendation list. MRR@5 is the reciprocal rank of the relevant item, where the reciprocal rank is manually set to zero if the rank is greater than 5.

Citation

@misc{quadrana2017personalizing,
  title={Personalizing Session-based Recommendations with Hierarchical Recurrent Neural Networks},
  author={Quadrana et al. (2017)},
  year={2017},
  note={arXiv:1706.04148}
}

• arXiv: 1706.04148