Openp5 Rec Eval

openp5-rec-eval

OpenP5: An Open-Source Platform for Developing, Training, and Evaluating LLM-based Recommender Systems — Xu et al. (2023) (arXiv:2306.11134, 2023)

What this evaluates

This benchmark evaluates the recommendation capability of LLM-based systems on sequential and straightforward recommendation tasks. It probes how well models leverage user interaction histories and different item indexing strategies to predict relevant items across multiple public datasets.

Datasets

• Movielens-1M — total ?; splits: test (-1)
• Amazon Beauty — total ?; splits: test (-1)
• LastFM — total ?; splits: test (-1)

Metrics

• HR@k (primary) — range: [0, 1]
  • Hit Ratio at rank k: the fraction of users for whom the ground-truth item appears in the top-k predicted items.
• NDCG@k (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank k: measures ranking quality by discounting the relevance of the ground-truth item by its position, normalized by the ideal ranking where the item is at rank 1.

Input / output format

Input: Text prompt containing user interaction history (item IDs) and a specific item indexing scheme (random, sequential, or collaborative), formatted for a T5-based LLM.

Output: Text string containing the predicted item ID(s) or a ranked list of items.

Scoring recipe

import math
def score_hr_ndcg(preds, gold, k=10):
    hr_vals, ndcg_vals = [], []
    for pred, true_id in zip(preds, gold):
        rank = pred.index(true_id) + 1 if true_id in pred else k + 1
        hr_vals.append(1.0 if rank <= k else 0.0)
        dcg = 1.0 / math.log2(rank + 1)
        idcg = 1.0 / math.log2(2)
        ndcg_vals.append(dcg / idcg)
    return sum(hr_vals)/len(hr_vals), sum(ndcg_vals)/len(ndcg_vals)

Common pitfalls

• Failing to distinguish between 'seen' and 'unseen' prompts, which tests zero-shot generalization and yields different performance trends.
• Overlooking the impact of item indexing methods (Random, Sequential, Collaborative), which significantly alter model performance and must be reported separately.
• Assuming SP5 (Super P5) generalizes uniformly; it suffers from dataset imbalance and overfitting on smaller datasets (Beauty, LastFM) compared to larger ones (ML1M).

Evidence (verbatim from paper)

Specifically, we use the top-$k$ Hit Ratio (HR@$k$) and Normalized Discounted Cumulative Gain (NDCG@$k$) to evaluate performance, providing the results for HR@5,10, and NDCG@5,10.

Citation

@misc{xu2023openp5,
  title={OpenP5: An Open-Source Platform for Developing, Training, and Evaluating LLM-based Recommender Systems},
  author={Xu et al. (2023)},
  year={2023},
  note={arXiv:2306.11134}
}

• arXiv: 2306.11134