Fgnn Sbr Eval

fgnn-sbr-eval

Exploiting Cross-Session Information for Session-based Recommendation with Graph Neural Networks — Qiu et al. (2021) (arXiv:2107.00852, 2021)

What this evaluates

Evaluates session-based recommendation models on e-commerce clickstream data by predicting the next item in a session using graph neural networks and cross-session information.

Datasets

• Yoochoose1/64 — total ?; splits: train (369859), test (55898)
• Yoochoose1/4 — total ?; splits: train (5917746), test (55898)
• Diginetica — total ?; splits: train (719470), test (60858)

Metrics

• R@20 (primary) — range: percent
  • R@K = n_hit / N, where N is the number of test sequences and n_hit counts sequences where the correct item is in the top K.
• MRR@20 (primary) — range: percent
  • MRR@K = (1/N) * Σ(1/Rank(v_label)), assigning 0 if the item is not in top K. Higher scores indicate better ranking quality.

Input / output format

Input: A sequence of item IDs representing a user's click history within a session, truncated to length i.

Output: A ranked list of top-K candidate item IDs for the next click in the session.

Scoring recipe

def evaluate(predictions, golds, K=20):
    hits = 0
    mrr_sum = 0.0
    for pred, gold in zip(predictions, golds):
        if gold in pred[:K]:
            hits += 1
            mrr_sum += 1.0 / (pred.index(gold) + 1)
    N = len(predictions)
    return hits / N, mrr_sum / N

Common pitfalls

• The train/test split uses the most recent 1/64 or 1/4 of the training sequence as the test set, which is a non-standard holdout strategy for session-based recommendation.
• Aggressive filtering removes all sessions of length 1 and items appearing fewer than 5 times, altering the original data distribution and potentially inflating metrics.
• Evaluation assumes a single ground-truth next item per session step, ignoring multi-label recommendations or alternative valid next items in real-world clickstreams.

Evidence (verbatim from paper)

For each time, a recommender system can give out a few recommended items and a user would choose the first few of them. To keep the same setting as previous baselines, we mainly choose to use top-20 items to evaluate a recommender system and specifically, two metrics, i.e., R@20 and MRR@20. For more detailed comparison, top-5 and top-10 results are considered as well.

Citation

@misc{qiu2021exploiting,
  title={Exploiting Cross-Session Information for Session-based Recommendation with Graph Neural Networks},
  author={Qiu et al. (2021)},
  year={2021},
  note={arXiv:2107.00852}
}

• arXiv: 2107.00852