Multi Session Rec Eval

multi-session-rec-eval

Effectively Using Long and Short Sessions for Multi-Session-based Recommendations — Wang et al. (2022) (arXiv:2205.04366, 2022)

What this evaluates

Evaluates a model's ability to perform next-item recommendation by leveraging both current session context and historical multi-session information. It probes how well the model handles varying session lengths (short vs. long) and filters out noise from irrelevant historical sessions.

Datasets

• Delicious — total ?; splits: train (-1), val (-1), test (-1)
• Reddit — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Recall@5 — range: [0, 1]
  • Proportion of ground-truth next items that appear in the top-5 recommended items, averaged over all test queries.
• Recall@20 (primary) — range: [0, 1]
  • Proportion of ground-truth next items that appear in the top-20 recommended items, averaged over all test queries.
• MRR@5 — range: [0, 1]
  • Mean Reciprocal Rank at 5: average of 1/rank for the first relevant item in the top-5 list, or 0 if none appear.
• MRR@20 — range: [0, 1]
  • Mean Reciprocal Rank at 20: average of 1/rank for the first relevant item in the top-20 list, or 0 if none appear.

Input / output format

Input: An ordered sequence of item IDs representing the user's current session, optionally augmented with historical session sequences or long-term interest features.

Output: A ranked list of candidate item IDs (top-K) predicted for the next interaction.

Scoring recipe

def compute_metrics(preds, golds, k=20):
    recalls = []
    mrrs = []
    for pred, gold in zip(preds, golds):
        top_k = pred[:k]
        if gold in top_k:
            recalls.append(1.0)
            rank = top_k.index(gold) + 1
            mrrs.append(1.0 / rank)
        else:
            recalls.append(0.0)
            mrrs.append(0.0)
    return sum(recalls) / len(recalls), sum(mrrs) / len(mrrs)

Common pitfalls

• The train/val/test splits overlap (train: 0-80%, val: 70-80%, test: 80-100%), meaning validation and test sets share user history. This can leak information if historical sessions are not strictly partitioned.
• Sessions are split using a strict 3600-second time gap, which differs from standard SBR benchmarks that often use fixed session IDs or different gap thresholds.
• Item frequency (<10) and session length (<2, >20) filtering are applied iteratively, which can drastically reduce dataset size compared to raw downloads.

Evidence (verbatim from paper)

So we choose the widely used ranking metrics, i.e., Recall@20 and Recall@5, and Mean Reciprocal Rank MRR@20 and MRR@5 to evaluate the recommendation performance in the experiments. ... Then we divide the sessions of each user into training set, validation set and test set according to the proportion of 0 - 80%, 70 - 80% and 80 - 100%...

Citation

@misc{wang2022effectively,
  title={Effectively Using Long and Short Sessions for Multi-Session-based Recommendations},
  author={Wang et al. (2022)},
  year={2022},
  note={arXiv:2205.04366}
}

• arXiv: 2205.04366