Sten Social Temporal Rec Eval

sten-social-temporal-rec-eval

Extracting Attentive Social Temporal Excitation for Sequential Recommendation — Li et al. (2021) (arXiv:2109.13539, 2021)

What this evaluates

Evaluates sequential recommendation models that incorporate social influence and temporal dynamics. It probes the ability to predict the next item a user will interact with based on their historical behavior sequence, social connections, and event timestamps.

Datasets

• Delicious — total ?; splits: train (-1), val (-1), test (-1); repo https://grouplens.org/datasets/hetrec-2011/
• Yelp — total ?; splits: train (-1), val (-1), test (-1); repo https://www.yelp.com/dataset
• Ciao — total ?; splits: train (-1), val (-1), test (-1); repo https://www.cse.msu.edu/tangjili/datasetcode/truststudy.htm

Metrics

• Recall@10 (primary) — range: [0, 1]
  • Percentage of ground-truth relevant items appearing within the top-10 ranked list. Equivalent to hit ratio for next-item recommendation.
• Recall@20 — range: [0, 1]
  • Percentage of ground-truth relevant items appearing within the top-20 ranked list.
• NDCG@10 — range: [0, 1]
  • Standard ranking metric reflecting correlation and position. For next-item recommendation, formulated as NDCG = 1/log2(1+r_p), where r_p is the rank of the positive item.
• NDCG@20 — range: [0, 1]
  • Same as NDCG@10 but evaluated over the top-20 ranked list.
• MRR@10 — range: [0, 1]
  • Mean reciprocal rank of the target item. Calculated as the mean of the reciprocal of the target item's actual rank. If rank > 10, reciprocal rank is set to zero.
• MRR@20 — range: [0, 1]
  • Same as MRR@10 but truncates reciprocal rank to zero only if rank > 20.

Input / output format

Input: User interaction history (sequence of item IDs with timestamps), social graph (friend lists/relations), and implicit feedback labels (1 for observed interactions).

Output: A ranked list of candidate items for the next interaction.

Scoring recipe

def compute_metrics(predictions, ground_truth, k):
    # predictions: list of recommended item IDs
    # ground_truth: single target item ID
    hit = 1 if ground_truth in predictions[:k] else 0
    recall = hit
    
    if ground_truth in predictions:
        rank = predictions.index(ground_truth) + 1
        dcg = 1.0 / math.log2(1 + rank)
        ndcg = dcg
        mrr = 1.0 / rank if rank <= k else 0.0
    else:
        ndcg = 0.0
        mrr = 0.0
    return recall, ndcg, mrr

Common pitfalls

• Yelp dataset uses only the first 10% of data chronologically for experiments, unlike standard full-dataset splits.
• MRR@k truncates the reciprocal rank to 0 if the target item's rank exceeds k, deviating from standard MRR which averages over all ranks.
• NDCG formula provided is simplified for a single positive item per query, ignoring standard DCG normalization over top-k positions.
• All ratings/reviews are binarized to implicit feedback (1), discarding explicit rating magnitudes and requiring careful handling of negative samples.

Evidence (verbatim from paper)

We adopt several common evaluation metrics to evaluate the recommendation performance, including Recall, Normalized Discount Cumulative Gain (NDCG) and Mean Reciprocal Rank (MRR): Recall@k. It means the percentage of groundtruth relevant items appear within the top-k ranking list. Consider to next-item recommendation, it is equivalent to the hit ratio. NDCG@k. It is a standard ranking metric and reflects both the correlation and position for each recommended item. For next-item recommendation, it is formulated as NDCG = 1/log2(1+r_p), where r_p is the rank of the positive item. MRR@k. It takes the actual rank of target item into consideration and is calculated by the mean value of the reciprocal of target item’s actual rank. When the actual rank is out of k, the reciprocal rank is set to zero.

Citation

@misc{li2021extracting,
  title={Extracting Attentive Social Temporal Excitation for Sequential Recommendation},
  author={Li et al. (2021)},
  year={2021},
  note={arXiv:2109.13539}
}

• arXiv: 2109.13539