Context Tree Session Rec Eval

context-tree-session-rec-eval

Context Tree for Adaptive Session-based Recommendation — Mi et al. (2018) (arXiv:1806.03733, 2018)

What this evaluates

This benchmark evaluates session-based recommendation models by predicting the immediate next item in a user's interaction sequence. It probes the model's ability to capture sequential dependencies and adapt to evolving user preferences and new items in both static and continuously updating environments.

Datasets

• MOOC, News, and RecSys Challenge Datasets — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• HR@k (primary) — range: [0, 1]
  • Averaged hit rate (recall) indicating whether the ground-truth next item appears in the top-k recommended items. Calculated as the mean of 1 if gold in top-k else 0 across all test instances.
• MRR@k — range: [0, 1]
  • Averaged mean reciprocal rank of the ground-truth next item within the top-k recommended list. Calculated as the mean of 1/rank if gold in top-k else 0 across all test instances.

Input / output format

Input: A sequence of previously interacted items in a session (e.g., [item_1, item_2, ..., item_t]).

Output: A ranked list of k candidate items predicted as the next item to interact with.

Scoring recipe

def compute_metrics(predictions, golds, k=20):
    hr_scores, mrr_scores = [], []
    for pred_list, gold in zip(predictions, golds):
        top_k = pred_list[:k]
        if gold in top_k:
            hr_scores.append(1.0)
            rank = top_k.index(gold) + 1
            mrr_scores.append(1.0 / rank)
        else:
            hr_scores.append(0.0)
            mrr_scores.append(0.0)
    return {'HR@k': sum(hr_scores) / len(hr_scores), 'MRR@k': sum(mrr_scores) / len(mrr_scores)}

Common pitfalls

• Static evaluation uses a fixed train/test split where the test set only contains items seen during training, whereas adaptive evaluation updates the model continuously as new events arrive, introducing unseen items.
• Preprocessing rules differ by dataset domain: MOOC and News datasets retain duplicate items within sessions, while RecSys Challenge datasets remove duplicates and filter out items with fewer than 5 total views.
• In the adaptive setting, GRU4Rec is updated only after a full session completes rather than per-event, which may unfairly penalize it compared to nonparametric methods that update incrementally.

Evidence (verbatim from paper)

The recommendation model generates a ranked list of k items as prediction for each testing interaction event, and the evaluation metrics that we are going to use across later experiments are as follow: - HR@k: The averaged hit rates (recall) of having the desired items amongst the top-k recommended item lists. - MRR@k: The averaged mean reciprocal ranks of the desired items in top-k recommended item lists.

Citation

@misc{mi2018contexttree,
  title={Context Tree for Adaptive Session-based Recommendation},
  author={Mi et al. (2018)},
  year={2018},
  note={arXiv:1806.03733}
}

• arXiv: 1806.03733