Csrec Sequential Rec Eval

csrec-sequential-rec-eval

Learning Robust Sequential Recommenders through Confident Soft Labels — Shiguang Wu et al. (2023) (arXiv:2311.02446, 2023)

What this evaluates

This evaluation protocol assesses the ranking performance and robustness of sequential recommendation models trained with confident soft labels. It measures whether predicted item sequences align with actual user interactions and verifies if recommendations correspond to genuinely positive user preferences using explicit rating thresholds.

Datasets

• Last.FM — total 52551; splits: train (-1), val (-1), test (-1)
• Yelp — total 316354; splits: train (-1), val (-1), test (-1)
• Amazon Electronics — total 684449; splits: train (-1), val (-1), test (-1)
• Amazon Movies and TV — total 625813; splits: train (-1), val (-1), test (-1)

Metrics

• Recall@n (primary) — range: [0, 1]
  • Measures whether the ground-truth item occurs in the top-n positions of the recommendation list. Formula: Recall@n = (1/|U|) * sum_{u in U} 1(rank_u <= n), where rank_u is the rank of the ground-truth item and 1(.) is the indicator function.
• NDCG@n (primary) — range: [0, 1]
  • A weighted ranking metric that attaches higher importance to top positions. Computed as the discounted cumulative gain of the ground-truth item at its predicted rank, normalized by the ideal DCG.
• Recall+@n — range: [0, 1]
  • A filtered version of Recall that only evaluates users whose ground-truth item has an explicit rating r_u >= δ (δ=4). Formula: Recall+@n = (1 / sum_{u in U} 1(r_u >= δ)) * sum_{u in U} 1(r_u >= δ ∧ rank_u <= n).
• NDCG+@n — range: [0, 1]
  • Filtered version of NDCG that applies the same rating threshold (δ=4) to restrict evaluation to users with genuinely positive preferences.

Input / output format

Input: Fixed-length sequences of user-item interactions (binary implicit feedback). Short sessions are padded with tokens; long sessions are cut into sub-sessions of fixed length (20 for Last.FM, 10 for others).

Output: A ranked list of items from the full item catalog, ordered by predicted relevance score.

Scoring recipe

def compute_recall_at_k(predictions, gold_items, k):
    hits = 0
    for u in users:
        rank = predictions[u].index(gold_items[u])
        if rank < k:
            hits += 1
    return hits / len(users)

def compute_ndcg_at_k(predictions, gold_items, k):
    dcg = 0.0
    for u in users:
        rank = predictions[u].index(gold_items[u])
        if rank < k:
            dcg += 1.0 / math.log2(rank + 2)
    idcg = 1.0 / math.log2(2)
    return dcg / idcg

Common pitfalls

• Training uses only binary implicit feedback, but filtered metrics like Recall+ require explicit rating thresholds (δ=4) to verify robustness; applying them to datasets without ratings will fail.
• The leave-one-out split strictly assigns the last interaction to test and the second-to-last to validation; using random splits or swapping these roles breaks the sequential evaluation protocol.
• Recall+ and NDCG+ are not standard metrics; they must be computed only over users with ground-truth ratings >= 4, which changes the denominator and excludes noisy implicit interactions.

Evidence (verbatim from paper)

To evaluate the recommendation performance, we adopt the leave-one-out evaluation procedure. The last item in a sequence is left as the test sample, while the one but last item is used for validation. The remaining interactions are used as the training set. We use the full item set as the candidate set when performing the ranking. For evaluation, we use two ranking-based metrics: (i) Recall and (ii) Normalized Discounted Cumulative Gain (NDCG). Recall@ $n$ measures whether the ground-truth item occurs in the top- $n$ positions of the list of recommendations. NDCG is a weighted version that attaches higher importance to top positions.

Citation

@misc{wu2023learning,
  title={Learning Robust Sequential Recommenders through Confident Soft Labels},
  author={Shiguang Wu et al. (2023)},
  year={2023},
  note={arXiv:2311.02446}
}

• arXiv: 2311.02446