Empatheticdialogues Eval

empatheticdialogues-eval

Towards Empathetic Open-domain Conversation Models: a New Benchmark and Dataset — Rashkin et al. (2018) (arXiv:1811.00207, 2018)

What this evaluates

This benchmark evaluates a model's ability to generate or retrieve empathetic, relevant, and fluent responses to emotionally grounded personal stories. It probes how well dialogue systems can acknowledge and react to a speaker's feelings in open-domain conversations.

Datasets

• EmpatheticDialogues — total ?; splits: train (-1), test (-1); repo https://github.com/facebookresearch/EmpatheticDialogues

Metrics

• BLEU — range: [0, 100]
  • Standard n-gram precision score comparing the model's generated or retrieved response against the gold human response.
• p@1,100 — range: [0, 1]
  • Accuracy of retrieving the exact gold response when given a candidate pool of 100 randomly sampled test-set examples that includes the gold response.
• Perplexity — range: other
  • Exponential of the average negative log-likelihood of the gold response tokens under the generative model.
• Human Empathy/Relevance/Fluency (primary) — range: [1, 5]
  • Average Likert scale rating (1-5) from crowdsourced raters assessing whether the response acknowledges feelings, is on-topic, and is linguistically fluent.

Input / output format

Input: A personal story/context from a speaker, optionally augmented with predicted emotion or topic labels.

Output: A single response string, either generated autoregressively or retrieved from a candidate pool.

Scoring recipe

def score_model(predictions, golds, human_ratings):
    # Automated: BLEU
    bleu_scores = [compute_bleu(ref=[g], hyp=p) for p, g in zip(predictions, golds)]
    avg_bleu = mean(bleu_scores)
    
    # Automated: p@1,100 (for retrieval setups)
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    p_at_100 = correct / len(predictions)
    
    # Human: Likert averages (1-5 scale)
    empathy_avg = mean([r['empathy'] for r in human_ratings])
    relevance_avg = mean([r['relevance'] for r in human_ratings])
    fluency_avg = mean([r['fluency'] for r in human_ratings])
    
    return {
        'BLEU': avg_bleu,
        'p@1,100': p_at_100,
        'Empathy': empathy_avg,
        'Relevance': relevance_avg,
        'Fluency': fluency_avg
    }

Common pitfalls

• Automated metrics like BLEU do not reliably correlate with human judgments of empathy or dialogue quality.
• The p@1,100 metric includes the gold response in the candidate pool during evaluation, which differs from standard zero-shot retrieval inference.
• Fine-tuning on this dataset may improve ED performance but can degrade performance on other conversation corpora like DailyDialog or Reddit.

Evidence (verbatim from paper)

For both retrieval and generative systems, we compute BLEU scores (Papineni et al., 2002) for the model response, comparing against the gold label (the actual response), following the practice of earlier work in dialogue generation (Wen et al., 2015; Li et al., 2016a, b). For the generative systems, we additionally report perplexity of the actual gold response. For the retrieval-based systems, we further compute p@1,100, the accuracy of the model at choosing the correct response out of a hundred randomly selected examples in the test set. When we compute p@1,100, the actual response is included in the candidates, unlike inference from the retrieval systems for all other metrics, which only uses training utterances as candidates.

Citation

@misc{rashkin2018empatheticdialogues,
  title={Towards Empathetic Open-domain Conversation Models: a New Benchmark and Dataset},
  author={Rashkin et al. (2018)},
  year={2018},
  note={arXiv:1811.00207}
}

• arXiv: 1811.00207