Dutch Medical Dialogue Eval

dutch-medical-dialogue-eval

Generating High Quality Synthetic Data for Dutch Medical Conversations — Kuan et al. (2026) (arXiv:2604.09645, 2026)

What this evaluates

Evaluates the quality of synthetically generated Dutch medical dialogues across structural, lexical, and qualitative dimensions to assess conversational naturalness and domain-specific accuracy.

Datasets

• Synthetic Dutch Medical Dialogues — total 9; splits: test (9)

Metrics

• MSTTR (primary) — range: [0, 1]
  • Moving-Segment Type-Token Ratio using overlapping sliding windows to measure local lexical variation. The paper notes MATTR was also computed but MSTTR was reported.
• Alternation rate — range: [0, 1]
  • Proportion of turns that alternate between doctor and patient roles, calculated as alternating transitions divided by total turns.
• Role consistency — range: [0, 1]
  • Keyword matching overlap score between role-specific lexicons (doctor/patient) and the generated dialogue text.
• Topic coverage — range: [0, 1]
  • Proportion of target clinical topics covered in the dialogue, measured via keyword matching against a predefined topic list.
• Human qualitative score — range: [1, 5]
  • Rater scores on a 5-point scale across five categories: Coherence, Consistency, Relevance, Clinical Use, and Fluency.

Input / output format

Input: Prompt containing a clinical topic/scenario, two-shot examples of real Dutch medical conversations, and structural constraints (e.g., one-sentence-per-turn).

Output: Multi-turn Dutch dialogue text alternating between doctor and patient roles.

Scoring recipe

def evaluate_dialogue(dialogue_text, gold_topics, doctor_lexicon, patient_lexicon):
    turns = split_into_turns(dialogue_text)
    sentences = split_into_sentences(dialogue_text)
    words = tokenize(dialogue_text)
    
    # Structural metrics
    roles = detect_roles(turns)
    alternation_rate = sum(roles[i] != roles[i-1] for i in range(1, len(roles))) / (len(roles) - 1)
    asl = len(sentences) / len(turns)
    spt = len(sentences) / len(turns)
    
    # Lexical & Role metrics
    ttr = len(set(words)) / len(words)
    msttr = moving_average_ttr(words, window_size=50)
    role_consistency = (count_keyword_matches(turns[::2], doctor_lexicon) + count_keyword_matches(turns[1::2], patient_lexicon)) / (len(turns) * max_vocab)
    topic_coverage = count_keyword_matches(dialogue_text, gold_topics) / len(gold_topics)
    
    return {
        'alternation_rate': alternation_rate,
        'role_consistency': role_consistency,
        'asl': asl,
        'spt': spt,
        'topic_coverage': topic_coverage,
        'ttr': ttr,
        'msttr': msttr
    }

Common pitfalls

• Automatic metrics like MSTTR and alternation rate correlate poorly with human ratings of conversational naturalness, capturing pattern regularity rather than semantic/pragmatic quality.
• Keyword-based role consistency and topic coverage fail to capture semantic correctness or contextual appropriateness of domain-specific vocabulary.
• Small sample size (n=9) and low inter-rater reliability (Krippendorff's α < 0.12) limit statistical generalizability and highlight rubric ambiguity.

Evidence (verbatim from paper)

Correlation analysis (Figure 12) confirmed weak alignment between automatic and human assessments: fluency and clinical use correlated moderately with MSTTR and role consistency, whereas relevance showed a negative relation (ρ = –0.31). Given the limited number of dialogues, these correlations should be interpreted with caution. These discrepancies highlight that numeric metrics capture pattern regularity but not semantic or pragmatic naturalness.

Citation

@misc{kuan2026generating,
  title={Generating High Quality Synthetic Data for Dutch Medical Conversations},
  author={Kuan et al. (2026)},
  year={2026},
  note={arXiv:2604.09645}
}

• arXiv: 2604.09645