Persona Chat Eval

persona-chat-eval

Personalizing Dialogue Agents: I have a dog, do you have pets too? — Zhang et al. (2018) (arXiv:1801.07243, 2018)

What this evaluates

Evaluates a dialogue agent's ability to generate or rank contextually appropriate next utterances while maintaining consistency with a given personal profile. It probes the model's capacity for persona-conditioned chit-chat and its ability to infer or reflect speaker interests during conversation.

Datasets

• PersonaChat — total 162064; splits: train (-1), test (-1)

Metrics

• Hits@1 (primary) — range: [0, 1]
  • Fraction of test instances where the correct next utterance appears as the top-ranked candidate among a set of options. For generative models, it measures whether the generated response exactly matches the ground truth or a candidate in the ranking pool.
• Perplexity — range: [0, inf)
  • Exponential of the average negative log-likelihood of the ground truth next utterance under the model's probability distribution. Lower values indicate better predictive performance.
• Human Ratings (Fluency, Engagingness, Consistency) — range: [1, 5]
  • Crowdsourced workers rate model responses on a 1-5 Likert scale across three dimensions after conversing with the agent. Higher scores indicate better perceived dialogue quality.
• Persona Detection Accuracy — range: [0, 1]
  • Binary accuracy measuring whether a human conversational partner correctly identifies the model's assigned persona from two candidate profiles after a dialogue.

Input / output format

Input: Dialogue history concatenated with the persona profile(s) of the speaker(s) to be conditioned on.

Output: For generative models: a single natural language next utterance. For ranking models: a ranked list of candidate responses.

Scoring recipe

def compute_hits_at_1(predictions, candidates, ground_truth):
    correct = 0
    for pred, gt in zip(predictions, ground_truth):
        if pred == gt or gt in candidates:
            correct += 1
    return correct / len(predictions)

# Human metrics are averaged 1-5 Likert scores across raters for fluency, engagingness, and consistency.
# Persona detection accuracy is the fraction of correct binary choices.

Common pitfalls

• Automated metrics like perplexity and hits@1 are notoriously poor proxies for actual dialogue quality and engagement.
• Word-overlap-based evaluation can be inflated by trivial responses that match persona keywords without semantic coherence.
• Human evaluation scores exhibit high variance, making subtle model differences hard to detect without large sample sizes.

Evidence (verbatim from paper)

Overall, the results show the following key points: Persona Conditioning Most models improve significantly when conditioning prediction on their own persona at least for the original (non-revised) versions, which is an easier task than the revised ones which have no word overlap. For example, the Profile Memory generation model has improved perplexity and hits@1 compared to Seq2Seq, and all the ranking algorithms (IR baseline, Starspace and Profile Memory Networks) obtain improved hits@1.

Citation

@misc{zhang2018personalizing,
  title={Personalizing Dialogue Agents: I have a dog, do you have pets too?},
  author={Zhang et al. (2018)},
  year={2018},
  note={arXiv:1801.07243}
}

• arXiv: 1801.07243