Sbr Intent Eval

sbr-intent-eval

Session-Based Recommendation with Validated and Enriched LLM Intents — Lee et al. (2025) (arXiv:2508.00570, 2025)

What this evaluates

Evaluates a session-based recommendation model's ability to predict the next item in a user session using validated and enriched LLM-generated intents. It probes the model's capacity to leverage semantic intent signals alongside sequential interaction patterns for accurate item ranking.

Datasets

• Beauty (Amazon) — total 7524; splits: train (6448), val (611), test (465)
• Yelp — total 41892; splits: train (34296), val (3985), test (3611)
• Books (Amazon) — total 20102; splits: train (16864), val (1723), test (1515)

Metrics

• Hit Rate@10 (primary) — range: [0, 1]
  • Fraction of test sessions where the ground-truth next item appears in the top-10 predicted items. Computed as the mean over all test sessions.
• NDCG@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at cutoff 10. Ranks the ground-truth item by its position in the predicted list, applying a logarithmic discount. Averaged over all test sessions.

Input / output format

Input: A sequence of item IDs representing a user's session history, optionally augmented with item metadata or LLM-derived intent embeddings.

Output: A ranked list of candidate items (typically top-20 for evaluation), from which the ground-truth next item's rank is extracted.

Scoring recipe

def compute_metrics(predictions, ground_truth, k=10):
    hits = 0
    ndcg_sum = 0.0
    for pred_list, true_item in zip(predictions, ground_truth):
        rank = pred_list.index(true_item) + 1 if true_item in pred_list else k + 1
        if rank <= k:
            hits += 1
            ndcg_sum += 1.0 / math.log2(1 + rank)
    hr = hits / len(ground_truth)
    ndcg = ndcg_sum / len(ground_truth)
    return hr, ndcg

Common pitfalls

• Splitting data by individual interaction timestamp instead of session timestamp causes severe data leakage in session-based recommendation.
• Averaging metrics over items instead of sessions, or vice versa, changes the evaluation scale significantly.
• Evaluating on top-5 or top-20 without specifying the cutoff leads to incomparable results.

Evidence (verbatim from paper)

We evaluate recommendation quality using Hit Rate (H) and NDCG (N) at cutoffs {5, 10, 20}, following prior studies*(Liu et al., [2024b]; Sun et al., [2024]; Ren et al., [2024]; Liu et al., [2023], [2024a])*. All results are averaged over five independent runs with different random seeds.

Citation

@misc{lee2025sessionbasedrecommendation,
  title={Session-Based Recommendation with Validated and Enriched LLM Intents},
  author={Lee et al. (2025)},
  year={2025},
  note={arXiv:2508.00570}
}

• arXiv: 2508.00570