Telemedicine Feedback Eval

telemedicine-feedback-eval

What Makes a Good Doctor Response? An Analysis on a Romanian Telemedicine Platform — Cosma et al. (2026) (arXiv:2602.17194, 2026)

What this evaluates

Predicts whether a patient will give positive feedback (thumbs-up) for a doctor's response in a Romanian telemedicine platform. It probes the model's ability to leverage clinical communication features, patient/doctor history, and metadata to forecast user satisfaction.

Datasets

• Romanian Telemedicine Platform Dataset — total 77334; splits: val (-1), val_high_conf (-1)

Metrics

• ROC-AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve; measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance across all thresholds.
• PR-AUC — range: [0, 1]
  • Area under the Precision-Recall curve; emphasizes performance on the positive class, especially useful for imbalanced datasets.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall; balances the trade-off between false positives and false negatives.
• Precision — range: [0, 1]
  • Ratio of true positive predictions to all positive predictions (true positives + false positives).
• Recall — range: [0, 1]
  • Ratio of true positive predictions to all actual positive instances (true positives + false negatives).

Input / output format

Input: Structured features combining doctor response text, patient history (e.g., prior thumbs-up rate, question count), doctor history (e.g., workplace, tenure, past feedback rates), and text-derived metrics (e.g., LIWC categories, character/token counts, number of digits/questions).

Output: Binary label indicating positive feedback (thumbs-up) or negative/missing feedback.

Scoring recipe

from sklearn.metrics import roc_auc_score, f1_score, precision_score, recall_score
y_true = [1 if fb == 'thumbs-up' else 0 for fb in gold]
preds = [1 if p >= 0.5 else 0 for p in predictions]
roc = roc_auc_score(y_true, predictions)
f1 = f1_score(y_true, preds)
prec = precision_score(y_true, preds)
rec = recall_score(y_true, preds)

Common pitfalls

• Text-level features contribute less to predictions than patient/doctor history and metadata, so models focusing solely on response text will underperform.
• Pearson correlations reported are modest (|r|≈0.02–0.15) and descriptive; they should not be interpreted as causal drivers of satisfaction.
• The 'high-confidence' validation subset is a filtered slice, not a standard held-out test set, and may overestimate real-world performance.

Evidence (verbatim from paper)

Since the classifier is properly calibrated (calibration error of 1.5%), high confidence predictions have high performance in terms of ROC-AUC score. Table 1 reports Feedback prediction performance across question subsets and patient tenure groups (New vs. Old), reported on the validation split and a high-confidence subset.

Citation

@misc{cosma2026good,
  title={What Makes a Good Doctor Response? An Analysis on a Romanian Telemedicine Platform},
  author={Cosma et al. (2026)},
  year={2026},
  note={arXiv:2602.17194}
}

• arXiv: 2602.17194