Regen Eval

regen-eval

REGEN: A Dataset and Benchmarks with Natural Language Critiques and Narratives — Su et al. (2025) (arXiv:2503.11924, 2025)

What this evaluates

Evaluates conversational recommender systems on next-item prediction and joint narrative generation, specifically testing how well models incorporate user interaction history and explicit natural language critiques to produce accurate recommendations and contextually grounded textual explanations.

Datasets

• REGEN — total ?; splits: Office Products (-1), Clothing (-1)

Metrics

• Recall@10 (primary) — range: [0, 1]
  • Fraction of ground-truth next items correctly predicted within the top-10 recommendations.
• NDCG@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10, measuring ranking quality with logarithmic position discounting.
• MRR — range: [0, 1]
  • Mean Reciprocal Rank, averaging the inverse rank of the first correct item across queries.
• BLEU — range: percent
  • Bilingual Evaluation Understudy score, measuring n-gram precision between generated and reference narratives.
• ROUGE — range: percent
  • Recall-Oriented Understudy for Gisting Evaluation, measuring n-gram recall/overlap between generated and reference narratives.
• Sem. Sim. — range: [0, 1]
  • Semantic Similarity, typically cosine similarity between sentence embeddings of generated and reference narratives.

Input / output format

Input: User interaction history sequence $S$ (item IDs, titles, descriptions, categories, and user-generated text) optionally augmented with explicit user critique commands $C$.

Output: A single item ID token (ID_TOKEN) followed by a natural language narrative $\mathcal{T}_{n+1}$.

Scoring recipe

def score(predictions, gold):
    rec_preds = [p[0] for p in predictions]
    rec_golds = [g[0] for g in gold]
    pred_texts = [p[1] for p in predictions]
    gold_texts = [g[1] for g in gold]
    recall = compute_recall_at_k(rec_preds, rec_golds, k=10)
    ndcg = compute_ndcg_at_k(rec_preds, rec_golds, k=10)
    mrr = compute_mrr(rec_preds, rec_golds)
    bleu = compute_bleu(pred_texts, gold_texts)
    rouge = compute_rouge(pred_texts, gold_texts)
    sem_sim = compute_semantic_similarity(pred_texts, gold_texts)
    return {'Recall@10': recall, 'NDCG@10': ndcg, 'MRR': mrr, 'BLEU': bleu, 'ROUGE': rouge, 'Sem. Sim.': sem_sim}

Common pitfalls

• Models are evaluated on two distinct architectures (hybrid vs. pure LLM) with different training regimes (separate vs. end-to-end), making direct comparison sensitive to training setup.
• Narrative generation is evaluated with an upper-bound oracle where the true next item ID is provided to the Flare+LLM baseline, which may inflate narrative quality compared to the autoregressive LUMEN model.
• Semantic similarity is reported without specifying the embedding model or distance metric used, making cross-study comparison difficult.

Evidence (verbatim from paper)

Table 4. Office Products Dataset - Benchmarks with Joint Recommendation and Generation Tasks. To evaluate upper bound performance, we provide the true next item $\mathrm{ID}_{t+1}$ when generating Flare+LLM narratives.

Citation

@misc{su2025regen,
  title={REGEN: A Dataset and Benchmarks with Natural Language Critiques and Narratives},
  author={Su et al. (2025)},
  year={2025},
  note={arXiv:2503.11924}
}

• arXiv: 2503.11924