Mentalchat16k Eval

mentalchat16k-eval

MentalChat16K: A Benchmark Dataset for Conversational Mental Health Assistance — Xu et al. (2025) (arXiv:2503.13509, 2025)

What this evaluates

Evaluates LLMs' capacity to generate empathetic, relevant, and contextually appropriate responses in mental health counseling scenarios. It probes the model's ability to demonstrate active listening, emotional validation, safety awareness, and ethical boundary adherence through a multi-dimensional rubric.

Datasets

• MentalChat16K — total 16000; splits: test (200); repo https://github.com/ChiaPatricia/MentalChat16K

Metrics

• MentalHealth Counseling Metrics (7 dimensions) (primary) — range: [0, 10]
  • Scores on 7 dimensions: Active Listening, Empathy & Validation, Safety & Trustworthiness, Open-mindedness & Non-judgment, Clarity & Encouragement, Boundaries & Ethical, Holistic Approach. Evaluated by GPT-4 Turbo, Gemini Pro, and human experts using a rubric. Average score is computed across the 7 dimensions.

Input / output format

Input: Counseling prompt/question from the test set (200 instances).

Output: Model-generated counseling response.

Scoring recipe

def compute_metric(predictions, gold_prompts, metrics):
    scores = []
    for prompt, response in zip(gold_prompts, predictions):
        for metric in metrics:
            score = llm_judge(prompt, response, metric) # or human_rater
            scores.append(score)
    return mean(scores)

Common pitfalls

• LLM evaluators exhibit strong alignment bias (GPT-4 favors synthetic data, Gemini values real interview data), making cross-model comparisons sensitive to the chosen judge.
• Human evaluation uses a ranking scale (1-7) rather than direct scoring, which may not correlate linearly with LLM-as-a-judge scores.
• The test set is small (200 questions), requiring multiple inference rounds and t-tests to establish statistical significance.

Evidence (verbatim from paper)

GPT 4’s evaluations reveal a consistent pattern favoring models fine-tuned on synthetic data (indicated by *). For example, in “Active Listening”, for all the seven base models, the fine-tuned version on synthetic data generated by GPT 3.5 Turbo outperforms the remaining three models including the base model, the model fine-tuned on the interview data and the model fine-tuned on both datasets.

Citation

@misc{xu2025mentalchat16k,
  title={MentalChat16K: A Benchmark Dataset for Conversational Mental Health Assistance},
  author={Xu et al. (2025)},
  year={2025},
  note={arXiv:2503.13509}
}

• arXiv: 2503.13509