Slmrec Eval

slmrec-eval

SLMRec: Distilling Large Language Models into Small for Sequential Recommendation — Wujiang Xu et al. (2024) (arXiv:2405.17890, 2024)

What this evaluates

Evaluates the sequential recommendation capability of a distilled small language model against traditional and LLM-based baselines. It measures ranking accuracy on user-item interaction histories and assesses computational efficiency (training/inference time and parameter count).

Datasets

• Amazon18 — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• HR@1 — range: [0, 1]
  • Binary indicator that is 1 if the ground-truth item appears at rank 1 in the predicted list, else 0.
• HR@5 — range: [0, 1]
  • Binary indicator that is 1 if the ground-truth item appears in the top-5 predicted items, else 0.
• NDCG@5 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 5. Computed as DCG@5 / IDCG@5, where DCG discounts the relevance by the logarithm of the rank position.
• MRR (primary) — range: [0, 1]
  • Mean Reciprocal Rank. The reciprocal of the rank position of the first relevant (ground-truth) item in the predicted list. Used for model selection on the validation set.

Input / output format

Input: A user's chronological interaction history (item IDs with timestamps). For evaluation, a candidate set of 1000 items is constructed: 1 ground-truth positive item and 999 randomly sampled negative items (items the user has not interacted with).

Output: A ranked list of the 1000 candidate items, ordered by predicted relevance scores.

Scoring recipe

def evaluate(ranked_candidates, ground_truth):
    hr1 = 1.0 if ranked_candidates[0] == ground_truth else 0.0
    hr5 = 1.0 if ground_truth in ranked_candidates[:5] else 0.0
    mrr = 0.0
    if ground_truth in ranked_candidates:
        mrr = 1.0 / (ranked_candidates.index(ground_truth) + 1)
    ndcg5 = 0.0
    if ground_truth in ranked_candidates[:5]:
        rank = ranked_candidates.index(ground_truth) + 1
        dcg = 1.0 / math.log2(rank + 1)
        idcg = 1.0 / math.log2(2)
        ndcg5 = dcg / idcg
    return hr1, hr5, ndcg5, mrr

Common pitfalls

• Evaluating on the full item catalog instead of the fixed 1000-candidate set (1 positive + 999 negatives) specified in the protocol.
• Selecting the best model based on test-set performance rather than validation-set MRR, which violates the stated unbiased evaluation methodology.
• Ignoring the strict chronological split (most recent for test, second most recent for val, preceding for train), which causes temporal data leakage if shuffled randomly.

Evidence (verbatim from paper)

The historical sequence of interactions for each user is divided into three segments: (1) the most recent interaction is reserved for testing, (2) the second most recent for validation, (3) all preceding interactions are used for training. Based on the ranking results, we utilize the typical top-N metrics hit rate (HR@{1, 5, 10}), normalized discounted cumulative gain (NDCG@{5,10}) and Mean Reciprocal Rank (MRR) to evaluate the model performance. ... we adopt the methodology employed in previous works, wherein we randomly select 999 negative items (i.e., items that the user has not interacted with) and combine with 1 positive item (i.e., a ground-truth interaction) to form our recommendation candidates for the ranking test.

Citation

@misc{xu2024slmrec,
  title={SLMRec: Distilling Large Language Models into Small for Sequential Recommendation},
  author={Wujiang Xu et al. (2024)},
  year={2024},
  note={arXiv:2405.17890}
}

• arXiv: 2405.17890