Dressipi Sbr Eval

dressipi-sbr-eval

A GNN Model with Adaptive Weights for Session-Based Recommendation Systems — Özbay et al. (2024) (arXiv:2408.05051, 2024)

What this evaluates

Evaluates a session-based recommendation model's ability to predict the next item a user will purchase based on their recent browsing history. It specifically probes how well the model handles cold-start scenarios and varying data availability by measuring ranking quality and hit rates on short retail sessions.

Datasets

• Dressipi — total ?; splits: 1/128 (28073), 1/64 (58244), 1/32 (112295), 1/4 (931919), 1/1 (3727678)

Metrics

• Recall@20 (primary) — range: percent
  • The proportion of test sessions where the ground-truth next item appears in the model's top-20 recommended list. Calculated as the average of binary hits (1 if in top-20, 0 otherwise) across all sessions.
• MRR@20 — range: [0, 1]
  • Mean Reciprocal Rank capped at 20. For each session, if the ground-truth item is ranked at position r ≤ 20, the score is 1/r; otherwise it is 0. Averaged across all sessions.

Input / output format

Input: A chronological sequence of item IDs representing a user's browsing session, optionally augmented with item features or side information (e.g., last item or session-averaged features).

Output: A ranked list of candidate items, typically truncated to the top-20 recommendations.

Scoring recipe

def compute_metrics(predictions, golds):
    recall_hits = 0
    mrr_sum = 0.0
    for pred_list, gold in zip(predictions, golds):
        if gold in pred_list[:20]:
            recall_hits += 1
            rank = pred_list[:20].index(gold) + 1
            mrr_sum += 1.0 / rank
    n = len(golds)
    return {
        'Recall@20': (recall_hits / n) * 100,
        'MRR@20': mrr_sum / n
    }

Common pitfalls

• Using different data fractions (e.g., 1/64 vs 1/1) drastically changes the cold-start dynamics and train/test item overlap, making direct metric comparisons across fractions misleading without context.
• MRR@20 explicitly sets the reciprocal rank to zero for any item ranked beyond position 20, which heavily penalizes models that push relevant items to lower ranks.
• Session length truncation (e.g., keeping only the last 5-20 items) is required for the adaptive weight mechanism to work effectively, but alters the natural session distribution.

Evidence (verbatim from paper)

Since recommender systems can only recommend a few items at a time, the actual item that the user can select should be among the first few items on the list. We therefore use the following two key metrics to evaluate the performance of recommendation lists. These metrics are commonly used to evaluate how effectively recommendation systems adapt to real-world use cases. • MRR@20: Mean Reciprocal Rank (MRR), which is the average of reciprocal ranks of the desired items. In the case of a rank higher than 20, the reciprocal rank is set to zero. MRR considers the ranking of each item, in scenarios where the sequence of recommendations is important. • Recall@20: The Recall@20 is the proportion of cases in which the desired item appears among the top 20 items. This metric plays a critical role in determining the effectiveness of recommendation systems.

Citation

@misc{ozbay2024gnn,
  title={A GNN Model with Adaptive Weights for Session-Based Recommendation Systems},
  author={Özbay et al. (2024)},
  year={2024},
  note={arXiv:2408.05051}
}

• arXiv: 2408.05051