Cotrec Eval

cotrec-eval

Self-Supervised Graph Co-Training for Session-based Recommendation — Xin Xia et al. (arXiv:2108.10560, 2021)

What this evaluates

Evaluates a model's ability to predict the next item in a user session based on historical click sequences. It probes the model's capacity to capture sequential dependencies and session-level patterns in sparse e-commerce or media interaction data.

Datasets

• Tmall — total ?; splits: train (-1), test (-1)
• RetailRocket — total ?; splits: train (-1), test (-1)
• Diginetica — total ?; splits: train (-1), test (-1)

Metrics

• P@10 (primary) — range: percent
  • Precision at K: the proportion of test instances where the ground-truth next item appears in the model's top-K recommended list.
• MRR@10 — range: percent
  • Mean Reciprocal Rank at K: the average of 1/rank for each test instance, where rank is the position of the ground-truth item in the top-K list (0 if not in top-K).

Input / output format

Input: A sequence of item IDs representing a user's session history (e.g., [item_1, item_2, ..., item_m]).

Output: A ranked list of top-K predicted item IDs for the next click.

Scoring recipe

def compute_metrics(predictions, golds, K=10):
    prec_scores = []
    mrr_scores = []
    for pred_list, gold in zip(predictions, golds):
        top_k = pred_list[:K]
        prec_scores.append(1.0 if gold in top_k else 0.0)
        if gold in top_k:
            rank = top_k.index(gold) + 1
            mrr_scores.append(1.0 / rank)
        else:
            mrr_scores.append(0.0)
    return {
        f'P@{K}': sum(prec_scores) / len(prec_scores) * 100,
        f'MRR@{K}': sum(mrr_scores) / len(mrr_scores) * 100
    }

Common pitfalls

• The dataset preprocessing splits each original session into multiple training/test sequences via a sliding window, which must be replicated exactly to avoid data leakage or mismatched evaluation counts.
• The paper reports metrics as percentages (e.g., 13.10), but standard implementations often output [0,1] decimals; ensure consistent scaling.
• Items appearing fewer than 5 times and sessions of length 1 are filtered out before splitting; omitting this step will inflate performance.

Evidence (verbatim from paper)

Following [38, 41], we use P@K (Precision) and MRR@K (Mean Reciprocal Rank) to evaluate the recommendation results where K is 10 or 20.

Citation

@misc{xia2021selfsupervised,
  title={Self-Supervised Graph Co-Training for Session-based Recommendation},
  author={Xin Xia et al.},
  year={2021},
  note={arXiv:2108.10560}
}

• arXiv: 2108.10560