Depression Diagnosis Chat Eval

depression-diagnosis-chat-eval

Enhancing Depression-Diagnosis-Oriented Chat with Psychological State Tracking — Gu et al. (2024) (arXiv:2403.09717, 2024)

What this evaluates

Evaluates a model's ability to conduct depression-diagnosis-oriented dialogues by tracking psychological states, generating appropriate responses, summarizing patient symptoms, and classifying depression/suicide severity. It also assesses conversational qualities like fluency, empathy, and doctor-likeness through human evaluation.

Datasets

• MedDialog — total ?; splits: train (-1), test (-1)

Metrics

• BLEU-2 (primary) — range: percent
  • Bigram overlap between generated and reference responses, computed after jieba tokenization.
• ROUGE-L — range: [0, 1]
  • Longest common subsequence overlap between generated and reference text.
• METEOR — range: [0, 1]
  • Precision, recall, and penalty for fragmentation between generated and reference text.
• DIST-2 — range: [0, 1]
  • Ratio of unique bigrams to total tokens in the generated response, measuring diversity.
• Average weighted F1 (primary) — range: [0, 1]
  • Weighted average of precision and recall across depression and suicide severity classes (2-class or 4-class), computed via sklearn.

Input / output format

Input: Dialogue history (and optionally patient portrait information) provided as context. For response generation, the model receives the dialogue context and psychological state tracking (POST) components (Stage, Information, Summary, Next) or operates without them.

Output: Generated dialogue response, patient symptom summary, or depression/suicide severity classification label (2-class or 4-class).

Scoring recipe

# Tokenize with jieba
preds_tok = [jieba.lcut(p) for p in predictions]
refs_tok = [jieba.lcut(r) for r in references]

# Response/Summary metrics
bleu2 = compute_bleu(n=2, preds=preds_tok, refs=refs_tok)
rougel = compute_rouge_l(preds=preds_tok, refs=refs_tok)
meteor = compute_meteor(preds=preds_tok, refs=refs_tok)
dist2 = sum(len(set(ngrams(t, 2))) for t in preds_tok) / sum(len(t) for t in preds_tok)

# Classification metrics
prec, rec, f1, _ = sklearn.metrics.precision_recall_fscore_support(y_true, y_pred, average='weighted')

Common pitfalls

• POSTs are evaluated in three modes: excluded ('-'), predicted, and given golden ('*'), which significantly impacts scores.
• Tokenization uses the jieba library for Chinese text, differing from standard spaCy/WordPiece tokenizers.
• ChatGPT baselines lack dataset-specific fine-tuning, leading to artificially lower BLEU/ROUGE scores compared to fine-tuned models.

Evidence (verbatim from paper)

Metrics like BLEU-2, Rouge-L and METEOR are employed to assess the response generation quality. In addition, we calculate DIST-2 to demonstrate the diversity of responses. Tokenization is performed using the jieba library... To be consistent with D4, we use average weighted precision, recall, and F1 scores computed by sklearn to evaluate classification results.

Citation

@misc{gu2024enhancing,
  title={Enhancing Depression-Diagnosis-Oriented Chat with Psychological State Tracking},
  author={Gu et al. (2024)},
  year={2024},
  note={arXiv:2403.09717}
}

• arXiv: 2403.09717