Session Aware Rec Eval

session-aware-rec-eval

Session-aware Recommendation: A Surprising Quest for the State-of-the-art — Latifi et al. (2020) (arXiv:2011.03424, 2020)

What this evaluates

Evaluates the predictive performance of session-based and session-aware recommendation models in ranking the next item a user will interact with. It benchmarks both neural and non-neural approaches across multiple real-world interaction datasets to assess accuracy, coverage, and popularity bias.

Datasets

• RETAIL — total ?; splits: test (-1)
• XING — total ?; splits: test (-1)
• COSMETICS — total ?; splits: test (-1)
• LASTFM — total ?; splits: test (-1)

Metrics

• MAP@20 (primary) — range: [0, 1]
  • Mean Average Precision at cutoff 20; averages precision scores at ranks where relevant items appear in the top-20 recommendations.
• P@20 — range: [0, 1]
  • Precision at 20; fraction of top-20 recommended items that are relevant.
• R@20 — range: [0, 1]
  • Recall at 20; fraction of all relevant items in the session that appear in the top-20 recommendations.
• HR@20 — range: [0, 1]
  • Hit Rate at 20; binary indicator (1 if at least one relevant item is in top-20, else 0).
• MRR@20 — range: [0, 1]
  • Mean Reciprocal Rank at 20; inverse of the rank of the first relevant item in the top-20 list.
• COV@20 — range: [0, 1]
  • Coverage at 20; fraction of distinct items recommended across the top-20 lists.
• POP@20 — range: other
  • Popularity at 20; average popularity (interaction frequency) of recommended items in the top-20 lists.

Input / output format

Input: A sequence of items interacted by a user in a session, optionally augmented with long-term user profile data for session-aware models.

Output: A ranked list of up to 20 recommended items.

Scoring recipe

def evaluate(preds, gold, k=20):
    hits = sum(1 for item in preds[:k] if item in gold)
    precision = hits / k
    recall = hits / len(gold) if gold else 0.0
    hr = 1.0 if hits > 0 else 0.0
    mrr = sum(1.0 / (i + 1) for i, item in enumerate(preds[:k]) if item in gold)
    ap = sum(precision for i, item in enumerate(preds[:k]) if item in gold) / min(len(gold), k)
    return {'P@20': precision, 'R@20': recall, 'HR@20': hr, 'MRR@20': mrr, 'MAP@20': ap}

Common pitfalls

• Using weak or improperly optimized baselines that fail to match the original authors' reported performance.
• Comparing session-based and session-aware models without properly accounting for long-term preference information or using inconsistent evaluation protocols.
• Neglecting statistical significance testing (e.g., Kruskal-Wallis and Wilcoxon signed-rank tests) when reporting performance differences.

Evidence (verbatim from paper)

Tables 3–6 show the results of our performance comparison of neural and non-neural methods, ordered by the values obtained for the MAP@20 metric. Here, we correspondingly report the values obtained by applying a cut-off threshold of 20.

Citation

@misc{latifi2020session,
  title={Session-aware Recommendation: A Surprising Quest for the State-of-the-art},
  author={Latifi et al. (2020)},
  year={2020},
  note={arXiv:2011.03424}
}

• arXiv: 2011.03424