Sequential Rec Eval

sequential-rec-eval

Pre-trained LLMs Meet Sequential Recommenders: Efficient User-Centric Knowledge Distillation — Severin et al. (2026) (arXiv:2604.21536, 2026)

What this evaluates

Evaluates the ability of sequential recommendation models to predict the next item in a user's interaction history by distilling semantic user profiles from pre-trained LLMs into the recommender's internal representations. The protocol tests whether knowledge distillation improves recommendation accuracy while maintaining inference efficiency without requiring real-time LLM calls.

Datasets

• Beauty — total 436309; splits: train (-1), test (-1)
• ML20M — total 19933088; splits: train (-1), test (-1)
• Kion — total 287698; splits: train (-1), test (-1)
• Amazon M2 — total 3651542; splits: train (-1), test (-1)

Metrics

• Recall@K, NDCG@K (primary) — range: [0, 1]
  • Standard sequential recommendation ranking metrics. Exact K values, tie-breaking conventions, and averaging methods are not specified in the provided text.

Input / output format

Input: Sequential list of user-item interactions (item IDs and timestamps) representing the user's historical behavior up to a temporal cutoff.

Output: Ranked list or probability distribution over candidate items for the next interaction.

Scoring recipe

# Metrics not explicitly named in the provided text.
# Standard sequential recommendation evaluation typically computes:
def compute_metrics(predictions, ground_truth, k_values=[10, 20]):
    scores = {}
    for k in k_values:
        pred_k = predictions[:k]
        scores[f'Recall@{k}'] = 1.0 if ground_truth in pred_k else 0.0
        scores[f'NDCG@{k}'] = 1.0 / math.log2(2) if ground_truth in pred_k else 0.0
    return scores

Common pitfalls

• Data leakage: Using future interactions for training or validation violates the temporal nature of sequential recommendation. The paper mitigates this with a strict global temporal split.
• k-core filtering: Removing users/items with fewer than 5 interactions significantly alters dataset density and size, making cross-dataset comparisons difficult without reporting filtered statistics.
• Hyperparameter selection: Grid search over many parameters with validation-based selection may lead to overfitting, especially on smaller datasets like Kion.

Evidence (verbatim from paper)

Datasets. We have evaluated our method on four publicly available datasets (Table 1) spanning different domains, scales, and levels of density. To ensure data quality, we applied k-core filtering with k=5 for all datasets, removing users and items with fewer interactions than this threshold. This cutoff was selected to balance data sparsity reduction with sufficient dataset coverage which is a standard practice. Following [14], we adopt a global temporal split (80/20) to avoid test leakage: interactions before the threshold form the training set, while later ones define the test set. All hyperparameters were selected based on performance on validation set, and final results are averaged over 5 runs with different random seeds.

Citation

@misc{severin2026pretrainedllmsmeet,
  title={Pre-trained LLMs Meet Sequential Recommenders: Efficient User-Centric Knowledge Distillation},
  author={Severin et al. (2026)},
  year={2026},
  note={arXiv:2604.21536}
}

• arXiv: 2604.21536