Lru Rec Eval

lru-rec-eval

Linear Recurrent Units for Sequential Recommendation — Yue et al. (2023) (arXiv:2310.02367, 2023)

What this evaluates

Evaluates sequential recommendation models on next-item prediction tasks across various domains (movies, products, games) with varying sequence lengths and sparsity.

Datasets

• ML-1M — total ?; splits: train (-1), val (-1), test (-1)
• Amazon-Beauty — total ?; splits: train (-1), val (-1), test (-1)
• Amazon-Video — total ?; splits: train (-1), val (-1), test (-1)
• Amazon-Sports — total ?; splits: train (-1), val (-1), test (-1)
• Steam — total ?; splits: train (-1), val (-1), test (-1)
• XLong — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Recall@10 (primary) — range: [0, 1]
  • Fraction of ground-truth items present in the top-10 recommended items.
• Recall@20 — range: [0, 1]
  • Fraction of ground-truth items present in the top-20 recommended items.
• NDCG@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10, weighting hits by their position in the ranked list.
• NDCG@20 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 20, weighting hits by their position in the ranked list.

Input / output format

Input: User interaction history sequence (item IDs) up to a dataset-specific maximum length (50, 200, or 1000).

Output: Ranked list of candidate items for the next interaction, evaluated against a single ground-truth item.

Scoring recipe

import math

def compute_recall_at_k(recommended_list, ground_truth, k):
    hits = sum(1 for item in recommended_list[:k] if item in ground_truth)
    return hits / len(ground_truth)

def compute_ndcg_at_k(recommended_list, ground_truth, k):
    dcg = 0.0
    for i, item in enumerate(recommended_list[:k]):
        if item in ground_truth:
            dcg += 1.0 / math.log2(i + 2)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(ground_truth), k)))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• The paper uses a leave-last-out split rather than random splitting, which is critical for sequential recommendation.
• During testing, validation items must be included in the candidate ranking pool, unlike standard IR benchmarks.
• XLong uses a reduced negative sampling (10k items) for evaluation efficiency, which may slightly bias metrics compared to full ranking.

Evidence (verbatim from paper)

For evaluation results, we select models with the best validation Recall@10 scores in training to perform prediction on the test sets. The models are evaluated using Recall@k and NDCG@k metrics, and with $k\in{10,20}$. The predicted items are ranked against all items in the dataset to compute the final scores.

Citation

@misc{yue2023lru,
  title={Linear Recurrent Units for Sequential Recommendation},
  author={Yue et al. (2023)},
  year={2023},
  note={arXiv:2310.02367}
}

• arXiv: 2310.02367