Senticxrl Eval

senticxrl-eval

SentiXRL: An advanced large language Model Framework for Multilingual Fine-Grained Emotion Classification in Complex Text Environment — Wang et al. (2024) (arXiv:2411.18162, 2024)

What this evaluates

Evaluates large language models on fine-grained emotion classification across English and Chinese dialogues and social media text. It probes the model's ability to handle complex, multilingual contexts, long sequences, and imbalanced emotion categories using a self-analytical negotiation mechanism.

Datasets

• MELD — total ?; splits: test (-1)
• EmoryNLP — total ?; splits: test (-1)
• IEMOCAP — total ?; splits: test (-1)
• CPED — total ?; splits: test (-1)
• CH-SIMS — total ?; splits: test (-1)
• Twitter2015 — total ?; splits: test (-1)
• Twitter2017 — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: percent
  • Ratio of correctly predicted emotion labels to the total number of instances.
• Macro-F1 — range: percent
  • Unweighted mean of per-class F1 scores, calculated as 2 * (precision * recall) / (precision + recall) for each class.
• Weighted-F1 — range: percent
  • Mean of per-class F1 scores weighted by the number of true instances for each class.

Input / output format

Input: Single-turn or multi-turn dialogue text (textual modality only), including context and target utterance.

Output: A discrete fine-grained emotion category label corresponding to the target utterance.

Scoring recipe

def compute_metrics(predictions, gold_labels):
    accuracy = sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    macro_f1 = f1_score(gold_labels, predictions, average='macro')
    weighted_f1 = f1_score(gold_labels, predictions, average='weighted')
    return {'accuracy': accuracy, 'macro_f1': macro_f1, 'weighted_f1': weighted_f1}

Common pitfalls

• Datasets exhibit significant class imbalance, which can skew accuracy and weighted metrics; macro-F1 is often preferred to evaluate minority class performance.
• The paper uses Weighted-F1 for English benchmarks but Macro-F1 for Chinese benchmarks, making direct cross-dataset metric comparison invalid without normalization.
• Multilingual datasets often rely on translated or projected annotations, which may not capture culture-specific emotional nuances and can limit cross-lingual generalization.

Evidence (verbatim from paper)

Specifically, in the Chinese sentiment classification task, our accuracy on the CPED dataset shows an improvement of 5.6% over the existing SOTA, and the F1 score on the CH-SIMS dataset increases by 1.55%. Similarly, in the English sentiment classification benchmarks, SentiXRL achieves the highest individual performance on the more challenging EmoryNLP dataset and surpasses the existing SOTA in the average Weighted-F1 score across three datasets.

Citation

@misc{wang2024senticxrl,
  title={SentiXRL: An advanced large language Model Framework for Multilingual Fine-Grained Emotion Classification in Complex Text Environment},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2411.18162}
}

• arXiv: 2411.18162