Sentiment Reasoning Healthcare Eval

sentiment-reasoning-healthcare-eval

Sentiment Reasoning for Healthcare — Nguyen et al. (2024) (arXiv:2407.21054, 2024)

What this evaluates

Evaluates a model's ability to jointly classify sentiment (negative, neutral, positive) from healthcare transcripts and generate semantically coherent rationales explaining the classification. It probes multimodal sentiment analysis, explainable AI, and chain-of-thought reasoning in a clinical dialogue setting.

Datasets

• Sentiment Reasoning dataset — total 30000; splits: test (-1); repo https://github.com/leduckhai/Sentiment-Reasoning

Metrics

• accuracy (primary) — range: [0, 1]
  • The proportion of correctly predicted sentiment labels out of the total number of instances. Labels are mapped to 0 (NEGATIVE), 1 (NEUTRAL), and 2 (POSITIVE).
• class-wise F1 score — range: [0, 1]
  • Macro-averaged F1 score computed across the three sentiment classes (NEGATIVE, NEUTRAL, POSITIVE), balancing precision and recall for each class equally.
• ROUGE score — range: [0, 1]
  • Recall-oriented n-gram overlap metric measuring the lexical overlap between the generated rationale and the human-annotated reference rationale.
• BERTScore — range: [0, 1]
  • Contextual embedding similarity metric that computes precision, recall, and F1 between generated and reference rationales using a pre-trained BERT model to capture semantic nuances.

Input / output format

Input: Transcript text (derived from human speech or ASR) representing a healthcare conversation turn.

Output: A sentiment label (0, 1, or 2) followed by a text rationale explaining the sentiment. Rationales may follow standard, elaborated, or step-by-step CoT formats.

Scoring recipe

def compute_metrics(predictions, gold_labels, generated_rationales, reference_rationales):
    # Accuracy & F1
    preds = [int(p) for p in predictions]
    acc = sum(p == g for p, g in zip(preds, gold_labels)) / len(gold_labels)
    f1 = macro_f1_score(gold_labels, preds)
    # ROUGE & BERTScore
    rouge = rouge_l_score(generated_rationales, reference_rationales)
    bert = bertscore_score(generated_rationales, reference_rationales)
    return {'accuracy': acc, 'f1': f1, 'rouge': rouge, 'bertscore': bert}

Common pitfalls

• ASR transcripts have a high Word-Error-Rate (~29.6%), which can artificially depress classification accuracy and rationale quality compared to human transcripts.
• ROUGE and BERTScore measure lexical/semantic overlap but do not verify the logical correctness or clinical safety of the generated rationales.
• Models must strictly output numeric labels (0, 1, 2) rather than text strings; failure to enforce this format during inference will break accuracy calculation.

Evidence (verbatim from paper)

For Sentiment Classificationtask, we employ accuracy and class-wise F1 score. For Rationale Generation, we employ ROUGE (Recall-Oriented Understudy for Gisting Evaluation) score Lin ([2004]). Also, we employ BERTScore [Zhang et al] which captures the contextual and semantic nuances. BERTscore has shown to correlate well with human judgment.

Citation

@misc{nguyen2024sentimentreasoning,
  title={Sentiment Reasoning for Healthcare},
  author={Nguyen et al. (2024)},
  year={2024},
  note={arXiv:2407.21054}
}

• arXiv: 2407.21054