Session Based Rec Eval

session-based-rec-eval

Context-aware Session-based Recommendation with Graph Neural Networks — Zhang et al. (2023) (arXiv:2310.09593, 2023)

What this evaluates

Evaluates a model's ability to predict the next item in a user's shopping session based on sequential item interactions and cross-session collaborative signals. It probes how well the model captures dynamic user interests and leverages historical session data for accurate recommendations.

Datasets

• Diginetica — total 780328; splits: train (719470), test (60858)
• Tmall — total 377166; splits: train (351268), test (25898)
• Yoochoose1_64 — total 425757; splits: train (369859), test (55898)

Metrics

• P@20 (primary) — range: percent
  • Precision at 20: the fraction of relevant items among the top-20 recommended items.
• MRR@20 — range: percent
  • Mean Reciprocal Rank at 20: the average reciprocal rank of the first relevant item in the top-20 list.

Input / output format

Input: A sequence of item IDs representing a user's session history.

Output: A ranked list of the top-20 candidate items.

Scoring recipe

def evaluate(predictions, ground_truth, k=20):
    p_at_k = 0
    mrr_at_k = 0
    for pred_list, gt in zip(predictions, ground_truth):
        top_k = pred_list[:k]
        if gt in top_k:
            p_at_k += 1
            rank = top_k.index(gt) + 1
            mrr_at_k += 1.0 / rank
    n = len(predictions)
    return (p_at_k / n) * 100, (mrr_at_k / n) * 100

Common pitfalls

• Strict data filtering (items <5 occurrences, sessions of length 1) and temporal split must be applied exactly as in [8] to match reported numbers.
• Using the full Yoochoose dataset instead of the 1/64 temporal split specified in the paper will yield different results.
• Metrics are reported as percentages (e.g., 56.49) rather than decimals, which can cause confusion when comparing with other works.

Evidence (verbatim from paper)

Further, we adopt two widely used evaluation metrics in information retrieval: Precision (P@20) and Mean Reciprocal Rank (MRR@20) for evaluating the performance.

Citation

@misc{zhang2023cares,
  title={Context-aware Session-based Recommendation with Graph Neural Networks},
  author={Zhang et al. (2023)},
  year={2023},
  note={arXiv:2310.09593}
}

• arXiv: 2310.09593