Llamarec Eval

llamarec-eval

LlamaRec: Two-Stage Recommendation using Large Language Models for Ranking — Yue et al. (2023) (arXiv:2311.02089, 2023)

What this evaluates

This evaluation probes a model's ability to rank candidate items based on a user's sequential interaction history and item metadata. It measures how effectively the system retrieves and re-ranks relevant products or movies against a large candidate pool using standard recommendation metrics.

Datasets

• ML-100k — total ?; splits: train (-1), val (-1), test (-1)
• Beauty — total ?; splits: train (-1), val (-1), test (-1)
• Games — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• MRR@k — range: [0, 1]
  • Mean Reciprocal Rank at k. Computes the average of 1/rank for the first relevant item in the top-k predictions. If no relevant item is in top-k, score is 0.
• NDCG@k (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at k. Measures ranking quality by assigning higher scores to relevant items appearing earlier in the list, normalized by the ideal DCG.
• Recall@k — range: [0, 1]
  • Recall at k. Calculates the fraction of relevant (ground truth) items successfully retrieved within the top-k predictions.

Input / output format

Input: User interaction history (up to 20 items) and candidate item titles (truncated to 32 tokens), formatted into a structured prompt for the LLM.

Output: A probability distribution over the candidate items, derived directly from the LLM's logits via a verbalizer mapping, rather than autoregressive text generation.

Scoring recipe

def compute_metrics(ranked_candidates, ground_truth, k=10):
    relevant_in_topk = [cand for cand in ranked_candidates[:k] if cand == ground_truth]
    recall = len(relevant_in_topk) / 1.0
    if relevant_in_topk:
        rank = ranked_candidates.index(ground_truth) + 1
        mrr = 1.0 / rank
        ndcg = (1.0 / math.log2(rank + 1)) / (1.0 / math.log2(2))
    else:
        mrr = 0.0
        ndcg = 0.0
    return mrr, ndcg, recall
# Average across all test instances

Common pitfalls

• Ranking performance is only evaluated on the 'valid retrieval subset' where the ground truth item must be in the top-20 retrieved candidates, which can mask retrieval failures.
• Comparison with some LLM baselines (e.g., GPT4Rec) uses different query batching strategies (5 queries per forward pass vs. 1), making direct metric comparisons potentially unfair.
• The leave-one-out split means test set size is not fixed across users, and metrics are averaged over all users rather than per-user averages.

Evidence (verbatim from paper)

In our evaluation, we follow the leave-one-out strategy and in each data example, we use the last item for testing, the second last item for validation, and the rest items for training. The evaluation metrics are mean reciprocal rank (MRR@k), normalized discounted cumulative gain (NDCG@k) and recall (Recall@k) with $k \in [5, 10]$. We save the model with best validation scores for evaluation (Recall@10 for retrieval and NDCG@10 for ranking), where predictions are ranked against all items in the dataset.

Citation

@misc{yue2023llamarec,
  title={LlamaRec: Two-Stage Recommendation using Large Language Models for Ranking},
  author={Yue et al. (2023)},
  year={2023},
  note={arXiv:2311.02089}
}

• arXiv: 2311.02089