Seq Aware Rec Eval

seq-aware-rec-eval

Sequence-Aware Recommender Systems — Quadrana et al. (2018) (arXiv:1802.08452, 2018)

What this evaluates

This survey evaluates and categorizes methodologies for sequence-aware recommender systems, focusing on offline evaluation protocols, data partitioning strategies, and ranking metrics used to assess prediction accuracy and list quality. It highlights how temporal dependencies and session boundaries require specialized splitting and target definition compared to traditional matrix completion.

Datasets

• Amazon — total 143000000; splits: train (-1), test (-1)
• RecSys Chall. 2015 — total 34000000; splits: train (-1), test (-1)
• Delicious — total 60000; splits: train (-1), test (-1)

Metrics

• Precision (primary) — range: [0, 1]
  • Fraction of recommended items that are relevant (correctly predicted) out of the total number of recommended items.
• Recall — range: [0, 1]
  • Fraction of relevant items that are successfully recommended out of the total number of relevant items in the ground truth.
• MAP — range: [0, 1]
  • Mean Average Precision: average of precision values computed at the rank of each relevant item, averaged across all queries/users.
• MRR — range: [0, 1]
  • Mean Reciprocal Rank: average of the reciprocal of the rank of the first relevant item in the recommended list.
• NDCG — range: [0, 1]
  • Normalized Discounted Cumulative Gain: measures ranking quality by summing graded relevance values of items at each position, discounted logarithmically, and normalized by the ideal DCG.

Input / output format

Input: Ordered sequence of user events/actions (e.g., timestamped interactions or session history), optionally with item types or context.

Output: Ranked list of recommended items or predicted next action(s) for the given sequence.

Scoring recipe

def compute_precision_at_k(predictions, ground_truth, k):
    top_k = predictions[:k]
    relevant = set(ground_truth)
    hits = sum(1 for item in top_k if item in relevant)
    return hits / k

def compute_map(predictions_list, ground_truth_list):
    ap_scores = []
    for preds, truth in zip(predictions_list, ground_truth_list):
        relevant_count = 0
        precision_sum = 0.0
        for rank, item in enumerate(preds, 1):
            if item in truth:
                relevant_count += 1
                precision_sum += relevant_count / rank
        ap_scores.append(precision_sum / len(truth) if truth else 0.0)
    return sum(ap_scores) / len(ap_scores)

Common pitfalls

• Using random data splits instead of time-based or session-level splits, which causes data leakage from future events into training.
• Ignoring session boundaries when partitioning data, which breaks the sequential dependency assumption of the model.
• Evaluating only on classification metrics (Precision/Recall) when the task requires ranking quality, leading to misleading performance assessments.

Evidence (verbatim from paper)

The set of metrics used in the reviewed papers include Precision, Recall, Mean Average Rank (MAR), Mean Average Precision (MAP), Mean Reciprocal Rank (MRR), Normalized Discounted Cumulative Gain (NDCG) and the F1 metric. Typically, most of the ranking metric are highly correlated when evaluated with a fixed-size split and their choice does not largely affect the outcomes [24, 44].

Citation

@misc{quadrana2018sequence,
  title={Sequence-Aware Recommender Systems},
  author={Quadrana et al. (2018)},
  year={2018},
  note={arXiv:1802.08452}
}

• arXiv: 1802.08452