Caser Sequential Rec Eval

caser-sequential-rec-eval

Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding — Tang et al. (2018) (arXiv:1809.07426, 2018)

What this evaluates

Evaluates a model's ability to capture sequential user behavior patterns for personalized top-N item recommendation. It probes the model's capacity to model temporal dependencies, skip behaviors, and union-level sequential patterns from historical interactions to predict future items.

Datasets

• MovieLens — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/graytowne/caser
• Gowalla — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/graytowne/caser
• Foursquare — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/graytowne/caser
• Tmall — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/graytowne/caser

Metrics

• Precision@N — range: [0, 1]
  • Precision@N = |R ∩ R̂_1:N| / N, where R is the test set (last 20% of user's sequence) and R̂_1:N is the top-N predicted items. Averaged over all users.
• Recall@N — range: [0, 1]
  • Recall@N = |R ∩ R̂_1:N| / |R|, where R is the test set and R̂_1:N is the top-N predicted items. Averaged over all users.
• MAP (primary) — range: [0, 1]
  • Mean Average Precision. AP = (Σ_{N=1}^{|R̂|} Prec@N × rel(N)) / |R̂|, where rel(N)=1 if the N-th predicted item is in R. MAP is the average AP across all users.

Input / output format

Input: A user's chronological sequence of past item interactions, converted to implicit feedback (binary 1). The sequence is processed up to a maximum length L (Markov order).

Output: A ranked list of top-N predicted items for the user.

Scoring recipe

def compute_metrics(predictions, ground_truth, Ns=[1, 5, 10]):
    precisions, recalls, aps = [], [], []
    for pred, gold in zip(predictions, ground_truth):
        gold_set = set(gold)
        for N in Ns:
            top_n = set(pred[:N])
            precisions.append(len(top_n & gold_set) / N)
            recalls.append(len(top_n & gold_set) / len(gold))
        ap_sum = 0.0
        for k, item in enumerate(pred):
            if item in gold_set:
                ap_sum += len(set(pred[:k+1]) & gold_set) / (k + 1)
        aps.append(ap_sum / len(pred))
    return {
        'Precision@N': {N: np.mean([p for p in precisions]) for N in Ns},
        'Recall@N': {N: np.mean([r for r in recalls]) for N in Ns},
        'MAP': np.mean(aps)
    }

Common pitfalls

• Data is split chronologically per user (70% train, 10% val, 20% test), not randomly shuffled.
• Cold-start users and items are filtered out (minimum 5-15 interactions depending on dataset).
• All numeric ratings are binarized to implicit feedback (1), discarding rating magnitude.
• GRU4Rec is session-based (no user ID), making direct comparison to personalized models potentially unfair for long-term preference modeling.

Evidence (verbatim from paper)

As in [19, 21, 29, 32], we evaluate a model by Precision@N, Recall@N, and Mean Average Precision (MAP). Given a list of top N predicted items for a user, denoted R̂1:N, and the last 20% of actions in her/his sequence (i.e., denoted R (i.e., the test set), Precision@N and Recall@N are computed by Prec@N = |R ∩ R̂_1:N| / N, Recall@N = |R ∩ R̂_1:N| / |R|. We report the average of these values of all users. N∈{1,5,10} The Average Precision (AP) is defined by AP = (sum{N=1}^{|R̂|} Prec@N × rel(N)) / |R̂|, where rel(N) = 1 if the N-th item in R̂ is in R. The Mean Average Precision (MAP) is the average of AP for all users.

Citation

@misc{tang2018caser,
  title={Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding},
  author={Tang et al. (2018)},
  year={2018},
  note={arXiv:1809.07426}
}

• arXiv: 1809.07426