Mentalbench Eval

mentalbench-eval

MentalBench: A Benchmark for Evaluating Psychiatric Diagnostic Capability of Large Language Models — Hoyun Song et al. (2026) (arXiv:2602.12871, 2026)

What this evaluates

Evaluates large language models' ability to perform psychiatric diagnostic decision-making using DSM-5 criteria. It probes their capacity to handle information incompleteness, perform differential diagnosis among overlapping disorders, and calibrate diagnostic commitment under varying prompt constraints.

Datasets

• MentalBench — total 24750; splits: test (24750); repo https://github.com/HoyunS/MentalBench

Metrics

• accuracy (exact match) (primary) — range: [0, 1]
  • Exact match accuracy. A prediction is scored as correct only if the model's selected diagnosis set exactly matches the complete set of ground-truth diagnoses; otherwise it is scored as incorrect.

Input / output format

Input: Synthetic clinical case descriptions (patient narratives or structured charts) varying in information completeness and diagnostic complexity, accompanied by instructions specifying whether to select a single diagnosis or one or more diagnoses.

Output: A list of psychiatric disorder names/codes (DSM-5) representing the predicted diagnoses for the given case.

Scoring recipe

correct = 0
for pred, gold in zip(predictions, gold_labels):
    if set(pred) == set(gold):
        correct += 1
accuracy = correct / len(predictions)

Common pitfalls

• Models often fail to infer the correct answer cardinality (single vs. multiple diagnoses) without explicit prompt constraints, leading to over- or under-diagnosis.
• Performance degrades sharply for disorders underrepresented in pre-training data (e.g., Schizoaffective Disorder), revealing knowledge retention gaps rather than reasoning flaws.
• Proprietary models tend to underperform on ambiguous cases due to rigid decisiveness, while open-source models struggle with unambiguous cases by failing to enforce exclusionary rules.

Evidence (verbatim from paper)

We measure accuracy using an exact match metric, recognizing a prediction as correct only if it identifies the complete set of ground-truth answers.

Citation

@misc{song2026mentalbench,
  title={MentalBench: A Benchmark for Evaluating Psychiatric Diagnostic Capability of Large Language Models},
  author={Hoyun Song et al. (2026)},
  year={2026},
  note={arXiv:2602.12871}
}

• arXiv: 2602.12871