Agentrecbench Eval

agentrecbench-eval

AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems — Shang et al. (2025) (arXiv:2505.19623, 2025)

What this evaluates

This benchmark evaluates LLM-based agentic recommender systems across three scenarios: classic, evolving-interest, and cold-start recommendation. It probes the agents' ability to dynamically plan, utilize textual interaction environments, and adapt to user preference shifts or data sparsity using structured user/item profiles and reviews.

Datasets

• Amazon — total ?; splits: test (-1)
• GoodReads — total ?; splits: test (-1)
• Yelp — total ?; splits: test (-1)

Metrics

• Hit Rate@$N (primary) — range: [0, 1]
  • Measures the probability that the ground-truth positive item appears in the top-N ranked positions. Formally: HR@N = (1/|T|) * sum_{t in T} I(p_t in R_t^N), where T is the test set, p_t is the ground-truth item, and R_t^N is the top-N recommendations.

Input / output format

Input: Structured user profile (ID, review count, social connections, avg rating), item profile (ID, name, type, metadata, avg rating, review count), review history, and a candidate set of 20 items (1 ground-truth positive + 19 unobserved negatives).

Output: A ranked list of items from the candidate set, typically returning the top-N recommendations.

Scoring recipe

def hit_rate_at_n(predictions, ground_truth, n):
    top_n = predictions[:n]
    return 1.0 if ground_truth in top_n else 0.0

# Aggregate over test set T:
# HR@N = mean(hit_rate_at_n(preds_t, gold_t, n) for t in T)
# Evaluated at N in {1, 3, 5}

Common pitfalls

• Negative sampling is fixed to exactly 19 unobserved items per test instance, which may artificially inflate or deflate ranking difficulty compared to real-world candidate pools.
• Cold-start thresholds (m for users, n for items) are dataset-dependent and not explicitly standardized in the text, making cross-dataset comparison of cold-start performance difficult.
• The evaluation focuses solely on ranking accuracy (HR@N) and does not assess conversational coherence, tool-use latency, or multi-turn interaction quality.

Evidence (verbatim from paper)

We evaluate recommendation performance using ranking-based metrics with emphasis on Top-$N$ accuracy. Following standard evaluation protocols*[[5], [7]]*, each test instance consists of 20 candidate items: one ground-truth positive item sampled from the user’s interaction history and 19 negative items sampled from unobserved interactions. The primary metric is Hit Rate@$N$ (HR@$N$), measuring the probability that the ground-truth item appears in the top-$N$ ranked positions ($N\in{1,3,5}$). Formally: $\text{HR@}N=\frac{1}{|\mathcal{T}|}\sum_{t\in\mathcal{T}}\mathbb{I}(p_{t}\in\mathcal{R}_{t}^{N})$

Citation

@misc{shang2025agentrecbench,
  title={AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems},
  author={Shang et al. (2025)},
  year={2025},
  note={arXiv:2505.19623}
}

• arXiv: 2505.19623