Surge Sequential Rec Eval

surge-sequential-rec-eval

Sequential Recommendation with Graph Neural Networks — Chang et al. (2021) (arXiv:2106.14226, 2021)

What this evaluates

Evaluates a model's ability to predict the next item in a user's interaction sequence by leveraging long-term historical behavior and filtering out noise. It probes the model's capacity to handle varying sequence lengths and efficiently model dynamic user preferences over time.

Datasets

• Taobao — total 1471155; splits: train (-1), val (-1), test (-1)
• Kuaishou — total 14952659; splits: train (-1), val (-1), test (-1)

Metrics

• AUC — range: [0, 1]
  • Signifies the probability that the positive item sample’s score is higher than the negative item sample’s score, reflecting the classification model’s ability to rank samples.
• GAUC (primary) — range: [0, 1]
  • Performs a weighted average of each user’s AUC, where the weight is his number of clicks. It eliminates the bias between users and evaluates model performance with a finer granularity.
• MRR — range: [0, 1]
  • Mean reciprocal rank, which is the mean value of the inverse of the ranking of the first hit item.
• NDCG@2 — range: [0, 1]
  • Assigns higher scores to hits at higher positions in the top-K ranking list, which emphasizes that test items should be ranked as higher as possible. K is set to 2.

Input / output format

Input: A sequence of item IDs representing a user's historical click behavior, along with a target item to predict.

Output: A predicted relevance score for each candidate item, used to generate a ranked list.

Scoring recipe

def compute_metrics(pred_scores, true_items, user_ids, neg_items):
    # GAUC
    user_auc = {}
    for uid in user_ids:
        pos = pred_scores[uid][true_items[uid]]
        negs = pred_scores[uid][neg_items[uid]]
        user_auc[uid] = np.mean(pos > negs)
    gauc = np.average(list(user_auc.values()), weights=[len(negs) for negs in neg_items.values()])
    
    # MRR & NDCG@2
    mrr = 0.0
    ndcg2 = 0.0
    for uid in user_ids:
        ranked = np.argsort(-pred_scores[uid])
        rank = np.where(ranked == true_items[uid])[0][0] + 1
        mrr += 1.0 / rank
        
        top2 = ranked[:2]
        dcg = sum(1.0 / np.log2(r + 2) for r, item in enumerate(top2) if item == true_items[uid])
        idcg = 1.0 / np.log2(2)
        ndcg2 += dcg / idcg
    mrr /= len(user_ids)
    ndcg2 /= len(user_ids)
    return gauc, mrr, ndcg2

Common pitfalls

• Using random train/val/test splits instead of the strict chronological time-based split specified in the paper.
• Ignoring the 10-core filtering and minimum 10-interaction threshold, which drastically changes dataset sparsity.
• Computing AUC globally instead of per-user before weighting for GAUC, which misrepresents recommendation performance.

Evidence (verbatim from paper)

To evaluate the performance of each model, we use two widely adopted accuracy metrics including AUC and GAUC(Zhou et al., [2018]), as well as two ranking metrics MRR and NDCG. They are defined as follows, • AUC signifies the probability that the positive item sample’s score is higher than the negative item sample’s score, reflecting the classification model’s ability to rank samples. • GAUC performs a weighted average of each user’s AUC, where the weight is his number of clicks. It eliminates the bias between users and evaluates model performance with a finer granularity. • MRR is the mean reciprocal rank, which is the mean value of the inverse of the ranking of the first hit item. • NDCG@K assigns higher scores to hits at higher positions in the top-K ranking list, which emphasizes that test items should be ranked as higher as possible. In our experiments, we set K to 2, a widely-used setting in existing works.

Citation

@misc{chang2021sequential,
  title={Sequential Recommendation with Graph Neural Networks},
  author={Chang et al. (2021)},
  year={2021},
  note={arXiv:2106.14226}
}

• arXiv: 2106.14226