Pen4rec Eval

pen4rec-eval

PEN4Rec: Preference Evolution Networks for Session-based Recommendation — Dou Hu et al. (arXiv:2106.09306, 2021)

What this evaluates

Evaluates a model's ability to predict the next item in a session-based recommendation task by capturing evolving user preferences and mitigating preference drift over time.

Datasets

• Yoochoose — total ?; splits: train (-1), val (-1), test (-1)
• Diginetica — total ?; splits: train (-1), val (-1), test (-1)
• LastFM — total ?; splits: train (-1), val (-1), test (-1)
• PHEME — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• P@20 (primary) — range: percent
  • Precision at rank 20. Calculated as the fraction of test sessions where the ground-truth next item appears in the top 20 recommended items.
• MRR@20 — range: percent
  • Mean Reciprocal Rank at rank 20. Calculated as the average of 1/rank for each session, where rank is the position of the ground-truth item in the recommended list (capped at 20).

Input / output format

Input: A sequence of items representing a user's session history, typically ordered by timestamp.

Output: A ranked list of candidate items for the next interaction.

Scoring recipe

def compute_metrics(recommended_list, ground_truth, k=20):
    hits = 1 if ground_truth in recommended_list[:k] else 0
    precision = hits / k
    rank = recommended_list.index(ground_truth) + 1 if ground_truth in recommended_list else k + 1
    mrr = 1 / rank if rank <= k else 0
    return precision, mrr

Common pitfalls

• Ignoring the 'timeframe' field to sort session items sequentially degrades performance and causes score discrepancies with prior work.
• The benchmark reports P@20 rather than the more common P@10 or P@5, which can mislead comparisons if not noted.

Evidence (verbatim from paper)

The scores on Diginetical dataset differ from results reported in [9,11] because they did not sort the session items according to "timeframe" field, which ignores the sequential information. CSRM outperforms them under the P@20 metric on LastFM dataset. Fig. 3 shows results against the hyper-parameter k on three datasets. The left Y-axis refers to P@20 (%) and the right Y-axis refers to MRR@20 (%).

Citation

@misc{dou2021pen4rec,
  title={PEN4Rec: Preference Evolution Networks for Session-based Recommendation},
  author={Dou Hu et al.},
  year={2021},
  note={arXiv:2106.09306}
}

• arXiv: 2106.09306