Tg Redial Eval

tg-redial-eval

Towards Topic-Guided Conversational Recommender System — Zhou et al. (2020) (arXiv:2010.04125, 2020)

What this evaluates

Evaluates a conversational recommender system's ability to naturally transition topics, recommend relevant items, and generate coherent responses within a dialogue. It probes the model's capacity to leverage historical interactions, user profiles, and topic sequences to maintain semantic flow and recommendation accuracy.

Datasets

• TG-ReDial — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/RUCAIBox/TG-ReDial

Metrics

• NDCG@k (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank k. It measures the quality of the ranked list of recommended items by assigning higher scores to relevant items appearing earlier in the list, normalized by the ideal ranking.
• MRR@k — range: [0, 1]
  • Mean Reciprocal Rank at rank k. It computes the average of the reciprocal ranks of the first relevant item in the predicted list across all queries, capped at rank k.
• Hit@k — range: [0, 1]
  • Binary hit rate at rank k. It indicates whether the ground-truth topic or item appears within the top k predictions.
• PPL — range: other
  • Perplexity, which measures how well a probability model predicts a sample. Lower values indicate better fit to the ground truth responses.
• BLEU-1 — range: [0, 1]
  • BLEU score computed at the unigram level. It measures the precision of matching single words between the generated response and the reference, with a brevity penalty.

Input / output format

Input: Conversation history (sequence of utterances), user profile information, and historical item interactions. For the topic prediction sub-task, the target topic is also provided as input.

Output: A ranked list of candidate items (for recommendation), a predicted topic label (for topic prediction), or a generated text response (for response generation).

Scoring recipe

def compute_ndcg_at_k(predictions, gold, k=10):
    # predictions: list of scores for all candidate items
    # gold: set of relevant item IDs
    ranked = sorted(predictions, reverse=True)[:k]
    dcg = sum(1.0 / math.log2(i + 2) for i, item in enumerate(ranked) if item in gold)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(gold), k)))
    return dcg / idcg if idcg > 0 else 0.0

def compute_mrr_at_k(predictions, gold, k=10):
    ranked = sorted(predictions, reverse=True)[:k]
    for i, item in enumerate(ranked):
        if item in gold:
            return 1.0 / (i + 1)
    return 0.0

Common pitfalls

• BLEU scores may be misleading for conversational recommendation systems as they are easily affected by stopwords and do not capture semantic relevance well.
• Human evaluation for response generation uses a [0, 2] rating scale for Relevance, Fluency, and Informativeness, which differs from standard 5-point Likert scales.
• The dataset is semi-automatically constructed with enforced topic threads, so models may overfit to artificial topic transitions rather than natural dialogue flow.

Evidence (verbatim from paper)

Following (Kang and McAuley, 2018; Liu et al., 2016b), we adopt NDCG@ $k$ and MRR@ $k$ ( $k = 10, 50$ ) as evaluation metrics for ranking all the possible items.

Citation

@misc{zhou2020towards,
  title={Towards Topic-Guided Conversational Recommender System},
  author={Zhou et al. (2020)},
  year={2020},
  note={arXiv:2010.04125}
}

• arXiv: 2010.04125