Srbench Eval

srbench-eval

SRBench: A Comprehensive Benchmark for Sequential Recommendation with Large Language Models — Li et al. (2026) (arXiv:2604.09553, 2026)

What this evaluates

Evaluates sequential recommendation models across accuracy, fairness, stability, and efficiency dimensions. It tests whether models can correctly rank items based on user interaction history and assesses their robustness, bias, and computational cost.

Datasets

• Yelp — total ?; splits: test (-1)
• ML-100K — total ?; splits: test (-1)
• Beauty — total ?; splits: test (-1)

Metrics

• Recall@5 (primary) — range: [0, 1]
  • Standard recommendation metric: fraction of ground-truth items in the top-5 recommended list. Exact calculation follows standard recsys conventions.
• NDCG@5 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 5. Measures ranking quality with position-based discounting.
• ARP — range: other
  • Benchmark-specific fairness metric measuring average recommendation popularity exposure. Exact formula defined in SRBench documentation.
• ARQ — range: other
  • Benchmark-specific fairness metric measuring average recommendation quality exposure. Exact formula defined in SRBench documentation.
• ARQV — range: other
  • Benchmark-specific stability metric measuring recommendation quality variance across runs. Exact formula defined in SRBench documentation.
• ARR — range: other
  • Benchmark-specific stability metric measuring recommendation consistency/rank stability. Exact formula defined in SRBench documentation.
• ART(s) — range: other
  • Benchmark-specific efficiency metric measuring average recommendation time in seconds.

Input / output format

Input: Structured user-item interaction sequences converted into natural language prompts via a unified paradigm with role-playing and semantic enrichment. Few-shot mode uses the 5 most recent interactions ordered chronologically.

Output: Unstructured natural language text containing ranked item recommendations. Must be parsed by extracting numerical item IDs and aligning them chronologically.

Scoring recipe

def score(predictions, ground_truth):
    # predictions: list of item IDs from LLM output
    # ground_truth: list of actual next-item IDs
    top5 = predictions[:5]
    recall = sum(1 for item in top5 if item in ground_truth) / len(ground_truth)
    ndcg = compute_ndcg(top5, ground_truth, k=5)
    # Fairness/Stability/Efficiency computed per SRBench protocol
    return recall, ndcg

Common pitfalls

• LLMs may hallucinate non-existent item IDs, requiring strict ID verification against the catalog.
• Output instability persists even at temperature=0 due to transformer architecture, necessitating multiple runs and averaging.
• Popularity bias skews fairness metrics (ARP/ARQ), causing models to favor popular items over quality.
• Sequence length significantly impacts performance; few-shot (k=5) often outperforms full-length in accuracy/fairness/efficiency.

Evidence (verbatim from paper)

As shown in Figure 6, on same MovieLens dataset, Caser's Recall@5 tested by SRBench is 0.0637, which is very close to Caser's own proposed evaluation result (0.0632) (Tang & Wang, 2018). Above consistency demonstrates that SRBench is capable of accurately evaluating SR models.

Citation

@misc{lib2026srbench,
  title={SRBench: A Comprehensive Benchmark for Sequential Recommendation with Large Language Models},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2604.09553}
}

• arXiv: 2604.09553