Llmrec Eval

llmrec-eval

LLMRec: Benchmarking Large Language Models on Recommendation Task — Liu et al. (2023) (arXiv:2308.12241, 2023)

What this evaluates

Benchmarks off-the-shelf large language models on five recommendation tasks: rating prediction, sequential recommendation, direct recommendation, explanation generation, and review summarization. It probes both accuracy-driven prediction capabilities and natural language generation for explainability, revealing gaps between objective metric scores and human-perceived quality in recommendation contexts.

Datasets

• LLMRec Benchmark (includes Beauty dataset) — total ?; splits: test (-1); repo https://github.com/williamliujl/LLMRec

Metrics

• RMSE — range: other
  • Root Mean Squared Error: sqrt(mean((y_true - y_pred)^2)).
• MAE — range: other
  • Mean Absolute Error: mean(|y_true - y_pred|).
• HR@5 — range: [0, 1]
  • Hit Rate at K: fraction of test items appearing in the top-K predicted list.
• NDCG@5 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at K: measures ranking quality with position-based discounting.
• HR@10 (primary) — range: [0, 1]
  • Hit Rate at K: fraction of test items appearing in the top-K predicted list.
• NDCG@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at K: measures ranking quality with position-based discounting.
• ROUGE1 — range: [0, 1]
  • Recall of unigram overlap between generated and reference text.
• ROUGE2 — range: [0, 1]
  • Recall of bigram overlap between generated and reference text.
• ROUGEL — range: [0, 1]
  • Recall of longest common subsequence overlap between generated and reference text.

Input / output format

Input: Textual prompts containing user/item context or interaction history, requesting a numerical rating, a ranked list of items, or a natural language explanation/summary.

Output: Numerical value (rating), comma-separated or list of item IDs/names, or free-form natural language text.

Scoring recipe

def score(preds, golds, K=10):
    rmse = np.sqrt(np.mean((np.array(preds) - np.array(golds))**2))
    mae = np.mean(np.abs(np.array(preds) - np.array(golds)))
    hr = sum(1 for p, g in zip(preds, golds) if g in p[:K]) / len(preds)
    ndcg = compute_ndcg_at_k(preds, golds, k=K)
    rouge = rouge_score(golds, preds, types=['rouge1', 'rouge2', 'rougeL'])
    return {'RMSE': rmse, 'MAE': mae, 'HR@K': hr, 'NDCG@K': ndcg, 'ROUGE': rouge}

Common pitfalls

• Off-the-shelf LLMs lack item-specific exposure, causing poor accuracy in rating and sequential recommendation tasks.
• Models often fail to produce standard structured outputs following prompt requirements without supervised fine-tuning (SFT).
• Objective metrics like ROUGE/BLEU do not correlate well with human-perceived quality of generated explanations or summaries.

Evidence (verbatim from paper)

In the context of explainability-based tasks, it is evident that off-the-shelf models are progressively closing the gap with trained models and, in some cases, superseding base-line algorithms based on certain metrics, as clearly demonstrated in Table 1 and Table 2. For instance, in review summarization task, ChatGPT achieves 4.7177, 0.6924, and 4.2557 on ROUGE1, ROUGE2, and ROUGEL metrics, respectively, exceeding the performance of T0 and GPT-2 trained on Beauty dataset.

Citation

@misc{liu2023llmrec,
  title={LLMRec: Benchmarking Large Language Models on Recommendation Task},
  author={Liu et al. (2023)},
  year={2023},
  note={arXiv:2308.12241}
}

• arXiv: 2308.12241