Medical Dialogue Gen Eval

medical-dialogue-gen-eval

Semi-Supervised Variational Reasoning for Medical Dialogue Generation — Dongdong Li et al. (2021) (arXiv:2105.06071, 2021)

What this evaluates

This evaluation probes a model's ability to generate clinically accurate, fluent, and context-aware physician responses in multi-turn medical dialogues. It assesses both automatic language quality and semantic relevance, alongside human-rated fluency, knowledge correctness, and overall satisfaction.

Datasets

• KaMed — total ?; splits: train (57754), valid (3000), test (3000)
• MedDialog — total ?; splits: train (32723), valid (3000), test (3000)
• MedDG — total ?; splits: train (14864), valid (2000), test (1000)

Metrics

• BLEU-2 (primary) — range: [0, 1]
  • Standard BLEU score computed over 2-grams, combining precision with a brevity penalty to penalize overly short generations.
• ROUGE-2 — range: [0, 1]
  • Recall-oriented overlap of 2-grams between the generated response and the reference response.
• Distinct-2 — range: [0, 1]
  • The proportion of distinct 2-grams in the generated responses, measuring lexical diversity.
• Entity F1 — range: [0, 1]
  • Macro and micro average Precision, Recall, and F1 scores of predicted medical entities in the generated responses.
• Human EQ — range: [1, 5]
  • Overall quality rating on a 1-5 Likert scale provided by professional medical annotators, with model names masked.

Input / output format

Input: Multi-turn dialogue context consisting of patient utterances, optionally augmented with medical knowledge graph entities and triplets.

Output: A single natural language physician response string.

Scoring recipe

def compute_rouge2(gold, pred):
    gold_tokens = tokenize(gold)
    pred_tokens = tokenize(pred)
    gold_ngrams = set(ngrams(gold_tokens, 2))
    pred_ngrams = set(ngrams(pred_tokens, 2))
    if not gold_ngrams:
        return 0.0
    overlap = len(gold_ngrams & pred_ngrams)
    return overlap / len(gold_ngrams)

Common pitfalls

• Relying solely on BLEU/ROUGE scores, as the paper explicitly notes these have known shortcomings for dialogue generation quality.
• Failing to mask model names during human evaluation, which can introduce annotator bias and invalidate Flu/KC/EQ ratings.
• Ignoring the semi-supervised setup; KaMed and MedDialog lack annotations, so evaluation must account for models trained on unlabeled data.

Evidence (verbatim from paper)

To assess the language quality for the generated responses, we employ classical word-overlap based metrics, BLEU-2 (B@2) [42] and ROUGE-2 (R@2)[31], to measure performance. As shortcomings have been reported for using BLEU/ROUGE to measure dialogue generation [33], we also use Distinct-1 (D@1) and Distinct-2 (D@2) [27], where Distinct-n is defined as the proportion of distinct n-grams in generated responses.

Citation

@misc{li2021semisupervised,
  title={Semi-Supervised Variational Reasoning for Medical Dialogue Generation},
  author={Dongdong Li et al. (2021)},
  year={2021},
  note={arXiv:2105.06071}
}

• arXiv: 2105.06071