Cmu Doq Eval

cmu-doq-eval

A Dataset for Document Grounded Conversations — Zhou et al. (2018) (arXiv:1809.07358, 2018)

What this evaluates

Evaluates the quality of generated responses in document-grounded conversations, specifically measuring how well models leverage external document context to produce engaging and fluent multi-turn dialogue. It assesses both automatic language modeling metrics and human-perceived response quality.

Datasets

• CMU.DoG — total 4112; splits: train (-1), test (-1); repo https://github.com/festvox/datasets-CMU_DoG

Metrics

• Perplexity (primary) — range: other
  • Standard n-gram language model perplexity computed on generated test responses. Lower values indicate better fluency and language modeling performance.
• Engagement Preference Rate — range: percent
  • Percentage of times a model's response is chosen over the baseline in a pairwise comparison given a 1-utterance chat history, with majority vote from 3 annotators.
• Fluency Score — range: [1, 4]
  • Mean rating on a 4-point Likert scale (1=unreadable, 4=perfectly readable) assigned by 3 annotators per response.

Input / output format

Input: Chat history (1 utterance) and corresponding document section/context for model generation; for human evaluation, chat history is presented to annotators alongside model responses.

Output: Generated response/utterance for the model; for human evaluation, a preference label (SEQ, SEQS, or No Preference) or a fluency score (1-4).

Scoring recipe

def compute_metrics(predictions, golds):
    # Perplexity
    ppl = exp(-mean(log_prob(predictions, ngram_model)))
    
    # Engagement Preference Rate
    votes = [pairwise_compare(history, pred, gold) for pred, gold in zip(predictions, golds)]
    engagement_rate = sum(votes == 'preferred') / len(predictions)
    
    # Fluency Score
    fluency_scores = [fluency_rating(pred) for pred in predictions]
    fluency_score = mean(fluency_scores)
    
    return ppl, engagement_rate, fluency_score

Common pitfalls

• The engagement metric includes a 'No Preference' option, so win rates for individual models do not sum to 100%.
• Fluency scores are averaged across 3 unique annotators per response, not calculated from raw individual ratings.
• Perplexity is computed using a custom n-gram model trained specifically on the training set responses, not a standard pre-trained language model.

Evidence (verbatim from paper)

To automatically evaluate the fluency of the models, we use perplexity measure. We build a language model on the train set of responses using ngrams up to an order of $3^{3}$. The generated test responses achieve a perplexity of 21.8 for the SEQ model and 10.11 for the SEQS model. We also perform two kinds of human evaluations to evaluate the quality of predicted utterances - engagement and fluency. The workers were asked to evaluate the fluency of the generated response on a scale of 1 to 4, where 1 is unreadable and 4 is perfectly readable.

Citation

@misc{zhou2018dataset,
  title={A Dataset for Document Grounded Conversations},
  author={Zhou et al. (2018)},
  year={2018},
  note={arXiv:1809.07358}
}

• arXiv: 1809.07358