Repeatnet Session Rec Eval

repeatnet-session-rec-eval

RepeatNet: A Repeat Aware Neural Recommendation Machine for Session-based Recommendation — Pengjie Ren et al. (2018) (arXiv:1812.02646, 2018)

What this evaluates

Evaluates session-based recommendation models on predicting the next item in a user session. It probes the model's ability to capture sequential patterns and handle repeat consumption behaviors across e-commerce and music domains.

Datasets

• YOOCHOOSE — total ?; splits: train (5325971), val (591775), test (55898)
• DIGINETICA — total ?; splits: train (647532), val (71947), test (60858)
• LASTFM — total ?; splits: train (2690424), val (333537), test (338115)

Metrics

• Recall@k (primary) — range: percent
  • The proportion of test cases where the ground-truth next item appears in the top-k recommended items.
• MRR@k — range: percent
  • The average of reciprocal ranks of the ground-truth items, truncated to 0 if the rank exceeds k.

Input / output format

Input: A sequence of items visited by a user in a session, ordered by time.

Output: A ranked list of candidate items for the next interaction.

Scoring recipe

def compute_metrics(predictions, targets, k=20):
    recalls = []
    mrrs = []
    for pred_list, target in zip(predictions, targets):
        recalls.append(1.0 if target in pred_list[:k] else 0.0)
        rank = pred_list.index(target) + 1 if target in pred_list else k + 1
        mrrs.append(1.0 / rank if rank <= k else 0.0)
    return {
        f'Recall@{k}': sum(recalls) / len(recalls) * 100,
        f'MRR@{k}': sum(mrrs) / len(mrrs) * 100
    }

Common pitfalls

• Filtering criteria differ across datasets (e.g., session length bounds, minimum item frequency).
• MRR@k truncates reciprocal rank to 0 if the target item is ranked beyond k.
• Datasets are split identically to prior work (Li et al. 2017b), not randomly.

Evidence (verbatim from paper)

Recall@k: The primary evaluation metric is Recall@k, which is the proportion of cases when the desired item is amongst the top-k items in all test cases. MRR@k: Another used metric is MRR@k (Mean Reciprocal Rank), which is the average of reciprocal ranks of the desire items. The reciprocal rank is set to zero if the rank is larger than k.

Citation

@misc{ren2018repeatnet,
  title={RepeatNet: A Repeat Aware Neural Recommendation Machine for Session-based Recommendation},
  author={Pengjie Ren et al. (2018)},
  year={2018},
  note={arXiv:1812.02646}
}

• arXiv: 1812.02646