Perrecbench Eval

perrecbench-eval

Can Large Language Models Understand Preferences in Personalized Recommendation? — Tan et al. (2025) (arXiv:2501.13391, 2025)

What this evaluates

Evaluates whether LLMs can capture true personalized user preferences by ranking items or users in groups, while explicitly controlling for confounding factors like user rating bias and item quality. It probes the model's ability to perform comparative reasoning rather than simple rating prediction.

Datasets

• PerRecBench — total ?; splits: test (-1); repo https://github.com/TamSiuhin/PerRecBench

Metrics

• Kendall’s tau (primary) — range: [-1, 1]
  • Measures the rank correlation between the predicted ordering and the ground truth ordering. Values range from -1 (perfect inverse correlation) to 1 (perfect correlation), with 0 indicating no association.

Input / output format

Input: Grouped prompts containing user and item context, formatted for pointwise (single user), pairwise (two users), or listwise (multiple users) ranking tasks.

Output: A ranked list of users or items according to the model's predicted preference order.

Scoring recipe

def kendall_tau(pred, true):
    n = len(pred)
    concordant = discordant = 0
    for i in range(n):
        for j in range(i + 1, n):
            diff_pred = pred[i] - pred[j]
            diff_true = true[i] - true[j]
            if diff_pred * diff_true > 0:
                concordant += 1
            elif diff_pred * diff_true < 0:
                discordant += 1
    return (concordant - discordant) / (n * (n - 1) / 2)

Common pitfalls

• Pointwise ranking methods evaluate users in isolation, making it difficult for models to capture subtle preference differences compared to pairwise or listwise approaches.
• Larger model size does not consistently improve personalization performance, challenging standard scaling law assumptions.
• Traditional regression metrics like rating prediction show low correlation with actual personalized ranking ability, so high prediction scores do not guarantee good personalization.

Evidence (verbatim from paper)

Across 19 strong LLMs, performance on PerRecBench ranges from 0.02 to 0.18, within Kendall’s tau value range of $[-1,1]$. This indicates a low to moderate correlation between predictions and ground truth rankings.

Citation

@misc{tan2025perrecbench,
  title={Can Large Language Models Understand Preferences in Personalized Recommendation?},
  author={Tan et al. (2025)},
  year={2025},
  note={arXiv:2501.13391}
}

• arXiv: 2501.13391