Preference Discerning Eval

preference-discerning-eval

Preference Discerning with LLM-Enhanced Generative Retrieval — Paischer et al. (2024) (arXiv:2412.08604, 2024)

What this evaluates

This benchmark evaluates a model's ability to dynamically adapt to evolving user preferences by conditioning on natural language preferences inferred from interaction history. It probes recommendation accuracy, fine- and coarse-grained preference steering, sentiment following, and history consolidation across multiple e-commerce and gaming datasets.

Datasets

• Amazon Beauty — total ?; splits: train (-1), val (-1), test (-1)
• Amazon Sports and Outdoors — total ?; splits: train (-1), val (-1), test (-1)
• Amazon Toys and Games — total ?; splits: train (-1), val (-1), test (-1)
• Steam — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Recall@10 (primary) — range: [0, 1]
  • Binary relevance metric: 1 if the ground-truth item appears in the top-10 generated predictions, 0 otherwise. Averaged over all test instances.
• Recall@5 — range: [0, 1]
  • Binary relevance metric: 1 if the ground-truth item appears in the top-5 generated predictions, 0 otherwise. Averaged over all test instances.
• NDCG@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at k=10. For binary relevance, DCG@10 = 1/log2(2) if relevant in top-10 else 0. IDCG@10 = 1/log2(2). NDCG = DCG/IDCG.
• NDCG@5 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at k=5. Computed identically to NDCG@10 but restricted to the top-5 predictions.

Input / output format

Input: A single natural language user preference string concatenated with a sequential history of up to 20 previously interacted items.

Output: A generated item identifier (token or semantic ID) representing the predicted next item.

Scoring recipe

def compute_recall_at_k(predictions, gold, k):
    return 1.0 if gold in predictions[:k] else 0.0

def compute_ndcg_at_k(predictions, gold, k):
    if gold in predictions[:k]:
        dcg = 1.0 / math.log2(2)
    else:
        dcg = 0.0
    idcg = 1.0 / math.log2(2)
    return dcg / idcg if idcg > 0 else 0.0

# Average over test set
total_recall = sum(compute_recall_at_k(pred, gold, 10) for pred, gold in test_instances)
recall_10 = total_recall / len(test_instances)

Common pitfalls

• The benchmark evaluates five distinct axes (Recommendation, Fine-grained steering, Coarse-grained steering, Sentiment following, History consolidation) that require different preference conditions; mixing them up leads to invalid comparisons.
• Models are evaluated using a leave-last-out split where the penultimate item is validation and the last is test; using standard random splits will inflate performance.
• Sentiment following performance is measured using m@k rather than standard Recall/NDCG, and overall scores are typically an order of magnitude lower than recommendation tasks.

Evidence (verbatim from paper)

We evaluate our trained baselines using common retrieval metrics, including Recall (or Hit Rate), and Normalized Discounted Cumulative Gain (Järvelin & Kekäläinen, [2002], NDCG).

Citation

@misc{paischer2024preferencediscerning,
  title={Preference Discerning with LLM-Enhanced Generative Retrieval},
  author={Paischer et al. (2024)},
  year={2024},
  note={arXiv:2412.08604}
}

• arXiv: 2412.08604