Reddit Cssrs Screening Eval

reddit-cssrs-screening-eval

Evaluating Reasoning LLMs for Suicide Screening with the Columbia-Suicide Severity Rating Scale — Patil et al. (2025) (arXiv:2505.13480, 2025)

What this evaluates

This benchmark evaluates zero-shot large language models on their ability to classify suicide risk severity from Reddit posts using the clinically validated Columbia-Suicide Severity Rating Scale (C-SSRS). It probes the models' ordinal classification capabilities, intent detection, and alignment with human clinical annotations across seven severity levels.

Datasets

• Reddit r/SuicideWatch posts (C-SSRS labeled) — total ?; splits: test (-1)

Metrics

• F1-Score (primary) — range: [0, 1]
  • Weighted harmonic mean of precision and recall across all C-SSRS severity classes. Computed as the macro/weighted average to account for class imbalance.
• Quadratic Weighted Kappa (QWK) — range: [0, 1]
  • Ordinal agreement metric that weights disagreements by the squared distance between predicted and actual severity levels, penalizing errors between distant levels more heavily.
• Spearman’s Rank Correlation (SRC) — range: [-1, 1]
  • Non-parametric measure of rank-order correlation between predicted and ground-truth severity levels, assessing how well models preserve the ordinal gradient.
• MAE — range: other
  • Mean Absolute Error; average of the absolute differences between predicted and actual C-SSRS severity integers.
• MSE — range: other
  • Mean Squared Error; average of the squared differences between predicted and actual severity integers, emphasizing larger ordinal deviations.
• Accuracy — range: [0, 1]
  • Proportion of posts where the predicted severity level exactly matches the ground-truth C-SSRS label.

Input / output format

Input: Raw text of a single Reddit post from r/SuicideWatch.

Output: A single integer from 0 to 6 representing the C-SSRS severity level (0: No ideation, 1: Wish to be dead, 2: Non-active suicidal thoughts, 3: Active ideation without intent, 4: Active ideation with intent but no plan, 5: Plan with intent, 6: Actual attempt).

Scoring recipe

def compute_metrics(preds, gold):
    n = len(gold)
    acc = sum(p == g for p, g in zip(preds, gold)) / n
    f1 = f1_score(gold, preds, average='weighted')
    mae = sum(abs(p - g) for p, g in zip(preds, gold)) / n
    mse = sum((p - g)**2 for p, g in zip(preds, gold)) / n
    qwk = quadratic_weighted_kappa(gold, preds)
    src = spearmanr(gold, preds).correlation
    return {'accuracy': acc, 'f1': f1, 'mae': mae, 'mse': mse, 'qwk': qwk, 'src': src}

Common pitfalls

• Misclassifications predominantly occur between adjacent severity levels (e.g., 1-2, 3-4, 5-6) rather than random errors, making standard accuracy misleading without ordinal metrics.
• Models frequently confuse nuanced intent (e.g., hopelessness vs. explicit plan) and produce false positives on emotionally charged but non-suicidal posts, requiring careful thresholding or human review.
• Short or context-poor posts significantly increase misclassification rates, so dataset composition and post length distribution heavily influence reported performance.

Evidence (verbatim from paper)

The evaluation includes standard classification metrics (accuracy, precision, recall, F1-score) as well as regression-based error measures (MAE, MSE) and ordinal agreement metrics (Quadratic Weighted Kappa [QWK] and Spearman’s Rank Correlation [SRC]).

Citation

@misc{patil2025evaluating,
  title={Evaluating Reasoning LLMs for Suicide Screening with the Columbia-Suicide Severity Rating Scale},
  author={Patil et al. (2025)},
  year={2025},
  note={arXiv:2505.13480}
}

• arXiv: 2505.13480