LLM Recommender Eval

llm-recommender-eval

Beyond Utility: Evaluating LLM as Recommender — Jiang et al. (2024) (arXiv:2411.00331, 2024)

What this evaluates

Evaluates the performance and behavioral characteristics of Large Language Models when deployed as recommender systems. It probes traditional recommendation accuracy and novelty, alongside LLM-specific traits like history length sensitivity, candidate position bias, and hallucination rates.

Datasets

• Unspecified recommendation datasets (four datasets referenced in paper) — total ?; splits: train (-1), test (-1); repo https://github.com/JiangDeccc/EvaLLMasRecommender

Metrics

• HR (primary) — range: [0, 1]
  • Hit Rate: proportion of users for whom the ground-truth next item appears in the top-K recommended list.
• NDCG — range: [0, 1]
  • Normalized Discounted Cumulative Gain: measures ranking quality by discounting the relevance of the ground-truth item based on its position in the list.
• APLT@K — range: [0, 1]
  • Average Percentage of Long Tail Items: (1/|R|) * sum(|R_u ∩ Φ| / K), where Φ contains items in the bottom 80% by interaction frequency.
• Serendipity@K — range: [0, 1]
  • Proportion of recommended items that are correct but were not recommended by a Most Popular baseline: (1/|R|) * sum(1/K * sum(1(i ∉ R_u^pop)))
• CandDif_Acc — range: other
  • Candidate Position Bias: -log(1 - Acc(R_first)) - (-log(1 - Acc(R_random))), measuring accuracy variation when positive items are moved from first to random positions.
• Hallucination — range: [0, 1]
  • Proportion of recommended items that fail case/symbol-agnostic string matching against the official item title set I.

Input / output format

Input: Prompt containing task instruction, user interaction history (h_u), and a candidate item set (C_u,y) indexed by item titles (T).

Output: A ranked list of top-K item titles (R_u) selected from the provided candidate set.

Scoring recipe

def evaluate(preds, gold, item_titles, long_tail_set, pop_recs, k):
    hr = sum(1 for p, g in zip(preds, gold) if g in p[:k]) / len(preds)
    ndcg = compute_ndcg(preds, gold, k)
    aplt = sum(len(set(p[:k]) & long_tail_set) / k for p in preds) / len(preds)
    serendipity = sum(1 for p in preds for i in p[:k] if i not in pop_recs) / (len(preds) * k)
    cand_diff = -np.log(1 - acc_first) + np.log(1 - acc_random)
    hallucination = sum(1 for p in preds for i in p[:k] if i not in item_titles) / (len(preds) * k)
    return {'HR': hr, 'NDCG': ndcg, 'APLT@K': aplt, 'Serendipity@K': serendipity, 'CandDif_Acc': cand_diff, 'Hallucination': hallucination}

Common pitfalls

• LLMs exhibit strong candidate position bias, often favoring items at the front of the input list regardless of prompt instructions to sort randomly.
• Hallucination evaluation requires strict string matching (ignoring case/symbols) against the official item title set I, as minor formatting differences can falsely flag valid items as hallucinations.
• Small-sample evaluation requires a Kolmogorov-Smirnov (K-S) test validation to ensure the sampled test set distribution matches the full dataset before running metrics.

Evidence (verbatim from paper)

Utility primarily represents the accuracy of recommendations. We utilize two widely-used metrics - HR and NDCG. HR measures the proportion of users obtaining accurate recommendations, while NDCG also takes the ranking quality of the recommendation results into consideration.

Citation

@misc{jiang2024beyondutility,
  title={Beyond Utility: Evaluating LLM as Recommender},
  author={Jiang et al. (2024)},
  year={2024},
  note={arXiv:2411.00331}
}

• arXiv: 2411.00331