Cape Sr Eval

cape-sr-eval

A Contextual-Aware Position Encoding for Sequential Recommendation — Yuan et al. (2025) (arXiv:2502.09027, 2025)

What this evaluates

This evaluation protocol assesses the effectiveness of context-aware position encoding methods in sequential recommendation systems. It measures how well models rank a target item given a user's historical interaction sequence, testing the model's ability to capture temporal and semantic dependencies in user behavior.

Datasets

• AmazonElectronics — total 1689188; splits: train (-1), val (-1), test (-1); HF reczoo/AmazonElectronics
• KuaiVideo — total 3239534; splits: train (-1), val (-1), test (-1); HF reczoo/KuaiVideo
• AmazonBooks — total 1689188; splits: train (-1), val (-1), test (-1); HF reczoo/AmazonBooks

Metrics

• AUC (primary) — range: [0, 1]
  • Probability that a randomly chosen positive sample is ranked higher than a randomly chosen negative sample.
• gAUC — range: [0, 1]
  • AUC computed per user separately, then averaged with a weighted average across users.
• Logloss — range: [0, ∞)
  • Negative log-likelihood of the predicted probability for the true label.
• Recall@K — range: [0, 1]
  • Fraction of relevant items retrieved in the top-K recommendations.
• NDCG@K — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank K, measuring ranking quality with position decay.

Input / output format

Input: User interaction history represented as sequences of item IDs and category IDs, along with a target item ID to predict. Context length is capped at 100 (or 50 for AmazonBooks).

Output: A continuous prediction score (probability) for the target item, used for ranking against negative samples.

Scoring recipe

def compute_auc(y_true, y_pred):
    pos = y_pred[y_true == 1]
    neg = y_pred[y_true == 0]
    if len(pos) == 0 or len(neg) == 0: return 0.0
    return (pos[:, None] > neg[None, :]).mean()

def compute_gauc(y_true, y_pred, user_ids):
    user_auc, weights = [], []
    for uid in np.unique(user_ids):
        mask = user_ids == uid
        if mask.sum() < 2: continue
        user_auc.append(compute_auc(y_true[mask], y_pred[mask]))
        weights.append(mask.sum())
    return np.average(user_auc, weights=weights)

Common pitfalls

• Applying standard NLP position encodings (e.g., RoPE, CoPE) directly to sequential recommendation often degrades performance, especially on short contexts (<100 items).
• AUC/gAUC improvements on the third decimal place are reported as statistically significant, which is unusually high precision for recommendation benchmarks and may mask practical relevance.
• Metrics are averaged over 5 random seeds, but the paper does not specify the exact train/val/test split ratios or temporal ordering conventions used for the reczoo datasets.

Evidence (verbatim from paper)

We evaluate performance using gAUC (Zhou et al., [2018]), AUC (Guo et al., [2017]) and Logloss. AUC measures the probability of a positive sample being ranked higher than a randomly chosen negative one. gAUC is to compute the AUC for each user separately, and then make a weighted average for them. In the experiment with large scale model, the metrics are Recall@K (R@K) and NDCG@K (N@K).

Citation

@misc{yuan2025cape,
  title={A Contextual-Aware Position Encoding for Sequential Recommendation},
  author={Yuan et al. (2025)},
  year={2025},
  note={arXiv:2502.09027}
}

• arXiv: 2502.09027