Dialogstudio Response Eval

dialogstudio-response-eval

DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI — Jianguo Zhang et al. (2023) (arXiv:2307.10172, 2023)

What this evaluates

Evaluates a model's ability to generate task-oriented and knowledge-grounded dialogue responses in zero-shot and few-shot settings. It measures lexical overlap and unigram F1 against ground-truth responses to assess generalization across open-domain and multi-domain conversational tasks.

Datasets

• CoQA — total ?; splits: test (-1)
• MultiWOZ 2.2 — total ?; splits: test (-1)

Metrics

• ROUGE-L (primary) — range: percent
  • Measures the longest common subsequence (LCS) overlap between the generated response and the ground-truth response.
• F1 — range: percent
  • Unigram F1 overlap score between the prediction and ground-truth response, as defined by Miller et al. (2017).

Input / output format

Input: Instruction prompt containing dialogue history formatted with and tokens, optionally including or dialogue acts, followed by the user's latest utterance.

Output: A single text string representing the system's next dialogue response.

Scoring recipe

def compute_metrics(predictions, references):
    rouge_l_scores = [rouge_l_score(pred, ref) for pred, ref in zip(predictions, references)]
    f1_scores = [unigram_f1_score(pred, ref) for pred, ref in zip(predictions, references)]
    return {
        "ROUGE-L": sum(rouge_l_scores) / len(rouge_l_scores),
        "F1": sum(f1_scores) / len(f1_scores)
    }

Common pitfalls

• Data leakage: CoQA and MultiWOZ 2.2 must be strictly excluded from the pre-training dataset to ensure valid zero-shot evaluation.
• Prompt formatting: Models must follow the exact instruction template (e.g., 'Continue the dialogue as a task-oriented dialogue system called SYSTEM...') to match the evaluation setup.

Evidence (verbatim from paper)

We focus on zero-shot evaluation and report the ROUGE-L and F1 score (Miller et al., 2017), where ROUGE-L measures the longest common subsequence and F1 measures the Unigram F1 overlap between the prediction and ground-truth response.

Citation

@misc{zhang2023dialogstudio,
  title={DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI},
  author={Jianguo Zhang et al. (2023)},
  year={2023},
  note={arXiv:2307.10172}
}

• arXiv: 2307.10172