Sbr Srgi Eval

sbr-srgi-eval

Exploring Global Information for Session-based Recommendation — Ziyang Wang et al. (arXiv:2011.10173, 2020)

What this evaluates

Evaluates session-based recommendation models by predicting the next item in a user's session sequence. It probes the model's ability to capture sequential item transitions and leverage global item-transition patterns across sessions to improve ranking accuracy.

Datasets

• Diginetica — total 780328; splits: train (719470), test (60858)
• Tmall — total 377166; splits: train (351268), test (25898)
• Nowplaying — total 915128; splits: train (825304), test (89824)

Metrics

• P@20 (primary) — range: [0, 1]
  • P@N = n_hit / n_test, where n_hit is the number of target items in the top N recommended list and n_test is the total number of test instances.
• MRR@20 — range: [0, 1]
  • MRR@N = (1/n_test) * Σ(1/Rank(v_target)), where the reciprocal rank is set to zero if the rank exceeds N.

Input / output format

Input: A session sequence of items [s1, s2, ..., sn] representing a user's interaction history.

Output: A ranked list of top-20 candidate items predicted as the next item in the session.

Scoring recipe

def compute_metrics(predictions, test_items):
    p_hits = 0
    mrr_sum = 0.0
    n_test = len(test_items)
    for pred, true_item in zip(predictions, test_items):
        if true_item in pred[:20]:
            p_hits += 1
            rank = pred.index(true_item) + 1
            mrr_sum += 1.0 / rank
    p_at_20 = p_hits / n_test
    mrr_at_20 = mrr_sum / n_test
    return p_at_20, mrr_at_20

Common pitfalls

• Hyperparameters are tuned on a random 10% subset of the training set rather than a dedicated validation split.
• Sequence splitting creates overlapping training and test instances, which can inflate performance if not strictly separated.
• Datasets undergo strict filtering (sessions length 1 removed, items <5 occurrences removed, Tmall sessions >40 removed), differing from raw data distributions.

Evidence (verbatim from paper)

We adopt two widely used ranking based metrics: P@N and MRR@N by following previous work. P@N(Precision): The P@N score is typically used as a measure of accuracy. It represents the proportion of correctly recommended items in top N recommended item list, which is defined as: P@N = n_hit / n_test, where n_test denotes the number of test data and n_hit denotes the number of the target items appearing in of top N recommended items. MRR@N(Mean Reciprocal Rank): The MRR@N score is the average of reciprocal rank of the correctly-recommended items. The reciprocal rank is set to zero if the rank exceeds N, MRR@N = (1/n_test) Σ(1/Rank(v_target)). Here, we choose N=20 for both P@N and MRR@N, as recommendation systems should focus on top ranked items.

Citation

@misc{wang2020exploring,
  title={Exploring Global Information for Session-based Recommendation},
  author={Ziyang Wang et al.},
  year={2020},
  note={arXiv:2011.10173}
}

• arXiv: 2011.10173