Fgnn Session Rec Eval

fgnn-session-rec-eval

Rethinking the Item Order in Session-based Recommendation with Graph Neural Networks — Qiu et al. (2019) (arXiv:1911.11942, 2019)

What this evaluates

Evaluates session-based recommendation models by predicting the next item in a user's clickstream session. It probes the model's ability to capture latent, non-temporal transition patterns and item order within a session graph.

Datasets

• Yoochoose — total ?; splits: 1/64 (-1), 1/4 (-1)
• Diginetica — total ?; splits: test (-1)

Metrics

• R@20 (primary) — range: percent
  • Recall calculated over top-K items. R@K = n_hit / N, where N is the number of test sequences and n_hit counts how many desired items appear in the top K positions of the predicted ranking list.
• MRR@20 — range: percent
  • Mean Reciprocal Rank calculated over top-K items. MRR@K = (1/N) * sum(1/Rank(v_label)) for v_label in test sequences. The reciprocal is set to 0 if the desired item is not in the top K positions.

Input / output format

Input: A partial session sequence of items [v_s,0, ..., v_s,i-1] of length i (2 ≤ i ≤ n), where the last item v_s,i-1 is the ground-truth target to predict.

Output: A ranked list of top-K items (default K=20) predicted as the next item in the session.

Scoring recipe

def compute_metrics(predictions, gold_items, K=20):
    hits = 0
    reciprocal_ranks = []
    for pred_list, gold in zip(predictions, gold_items):
        if gold in pred_list[:K]:
            hits += 1
            rank = pred_list[:K].index(gold) + 1
            reciprocal_ranks.append(1.0 / rank)
        else:
            reciprocal_ranks.append(0.0)
    recall = (hits / len(gold_items)) * 100
    mrr = (sum(reciprocal_ranks) / len(gold_items)) * 100
    return recall, mrr

Common pitfalls

• Sessions are augmented by splitting each session of length n into n-1 partial sessions of lengths 2 to n, significantly inflating the dataset size compared to standard session-based splits.
• Datasets are pre-filtered to remove sessions of length 1 and items occurring fewer than 5 times, which differs from raw clickstream data and affects baseline comparability if not replicated.
• Metrics are reported as percentages in tables (e.g., 71.12) despite the mathematical formulas using a [0, 1] scale, which can cause confusion when comparing against other papers.

Evidence (verbatim from paper)

At a time, a recommender system can give out a few recommended items and a user would choose the first few of them. To keep the same setting as previous baselines, we mainly choose to use top-20 items to evaluate a recommender system and specifically, two metrics, i.e., R@20 and MRR@20. For more detailed comparison, top-5 and top-10 results are considered as well. • R@K (Recall calculated over top-K items). The R@K score is the primary metric that calculates the proportion of test cases which recommend the correct item in a top K position in a ranking list, R@K = n_hit / N, where N represents the number of test sequences S_test in the dataset and n_hit counts the number that the desired items are in the top K position in the ranking list, which is named the hit. • MRR@K (Mean Reciprocal Rank calculated over top-K items). The reciprocal is set to 0 when the desired items are not in the top K position and the calculation is as follows, MRR@K = 1/N sum_{v_label in S_test} 1/Rank(v_label).

Citation

@misc{qiu2019rethinking,
  title={Rethinking the Item Order in Session-based Recommendation with Graph Neural Networks},
  author={Qiu et al. (2019)},
  year={2019},
  note={arXiv:1911.11942}
}

• arXiv: 1911.11942