Psyeval Eval

psyeval-eval

MCTSr-Zero: Self-Reflective Psychological Counseling Dialogues Generation via Principles and Adaptive Exploration — Lu et al. (2025) (arXiv:2505.23229, 2025)

What this evaluates

Evaluates an AI model's ability to generate empathetic, principle-constrained psychological counseling dialogues in simulated multi-turn interactions. It probes competencies like accurate empathy, logical consistency, resistance handling, and ethical guidance beyond surface-level language features.

Datasets

• PsyEval — total ?; splits: test (-1)

Metrics

• PsyEval (primary) — range: other
  • Aggregate score across 16 predefined counseling dimensions (e.g., empathy, logical consistency, resistance handling, ethics). Each dimension is scored by an independent AI judge using standardized rubrics, then averaged to produce the final metric.

Input / output format

Input: Synthesized psychological case reports/scenarios prompting the model to generate a multi-turn counseling dialogue.

Output: Multi-turn psychological counseling dialogue generated by the AI therapist.

Scoring recipe

def compute_psyeval(dialogue, rubric_dimensions):
    dim_scores = []
    for dim in rubric_dimensions:  # 16 dimensions
        score = ai_judge.evaluate(dialogue, dim)
        dim_scores.append(score)
    return sum(dim_scores) / len(dim_scores)

Common pitfalls

• The evaluation relies entirely on an AI judge rather than human raters, which may introduce model-specific biases or alignment artifacts.
• The 16 dimensions are broad and theory-driven; scoring may vary significantly depending on the specific AI judge model and prompt configuration used.
• Synthetic case reports are used instead of real patient data, which may limit ecological validity compared to clinical benchmarks.

Evidence (verbatim from paper)

A central component of PsyEval is our novel, multi-dimensional evaluation benchmark, comprising 16 core assessment dimensions. It assesses an AI therapist’s ability to provide empathic support in multi-turn interactions... Evaluation within PsyEval is performed by an independent AI model configured according to our benchmark’s rubrics. This enables scalable, efficient, and consistent scoring across numerous simulated dialogues.

Citation

@misc{lu2025mctsrzero,
  title={MCTSr-Zero: Self-Reflective Psychological Counseling Dialogues Generation via Principles and Adaptive Exploration},
  author={Lu et al. (2025)},
  year={2025},
  note={arXiv:2505.23229}
}

• arXiv: 2505.23229