Convai2 Eval

convai2-eval

The Second Conversational Intelligence Challenge (ConvAI2) — Dinan et al. (2019) (arXiv:1902.00098, 2019)

What this evaluates

Evaluates open-domain chatbot capabilities in persona-driven multi-turn conversations, measuring response quality via automatic metrics and human judgments of engagement and persona consistency.

Datasets

• PERSONA-CHAT — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/DeepPavlov/convai

Metrics

• F1 — range: [0, 1]
  • Word overlap F1 score between the generated response and the ground truth response.
• Hits@1 — range: [0, 1]
  • Fraction of instances where the model ranks the correct response as the top candidate among a set of candidates.
• Perplexity — range: other
  • Exponential of the average negative log-likelihood of the ground truth response tokens under the model's language model distribution.
• Engagingness (primary) — range: [1, 4]
  • Human annotator rating on a 1-4 scale answering 'How much did you enjoy talking to this user?' after a 4-6 turn dialogue.
• Persona Detection — range: [0, 1]
  • Fraction of times the human correctly identifies the model's assigned persona versus a random distractor persona.

Input / output format

Input: Multi-turn conversation history with assigned personas for both human and model, followed by a prompt for the model to generate the next utterance.

Output: A single text response string.

Scoring recipe

def score(predictions, golds, candidates=None, human_ratings=None):
    # Automatic metrics
    f1_vals = [f1_score(g.split(), p.split()) for p, g in zip(predictions, golds)]
    hits1 = sum(1 for c, g in zip(candidates, golds) if c[0] == g) / len(golds)
    ppl = exp(-mean([log_prob(g) for g in golds]))
    # Human metrics
    engagingness = mean(human_ratings) if human_ratings else None
    persona_det = mean([1 if correct else 0 for correct in human_ratings]) if human_ratings else None
    return {'F1': mean(f1_vals), 'Hits@1': hits1, 'Perplexity': ppl, 'Engagingness': engagingness, 'Persona Detection': persona_det}

Common pitfalls

• F1 score is easily gamed by outputting frequent training words without coherent dialogue, as shown by a toy baseline outperforming all models.
• Automatic metrics (F1, Hits@1) correlate poorly with human engagement judgments, causing leaderboard rankings to misrepresent actual chatbot quality.
• Unpaid 'wild' evaluations yield noisy, off-instruction dialogues that lack systematic filtering criteria and were ultimately discarded.

Evidence (verbatim from paper)

The rank of each team was determined by sorting by the minimum rank of the score in any of the three metrics (F1, Hits@1, and Perplexity). ... For each evaluation, we paired a human worker with a model, assigned each of them personas, and instructed the humans to chat with and get to know their partner. Dialogues were of length 4-6 turns each. Following a short conversation, we asked workers "How much did you enjoy talking to this user?" and had them answer on a scale of 1-4.

Citation

@misc{dinan2019convai2,
  title={The Second Conversational Intelligence Challenge (ConvAI2)},
  author={Dinan et al. (2019)},
  year={2019},
  note={arXiv:1902.00098}
}

• arXiv: 1902.00098