Emotiontalk Eval

emotiontalk-eval

EmotionTalk: An Interactive Chinese Multimodal Emotion Dataset With Rich Annotations — Sun et al. (2025) (arXiv:2505.23018, 2025)

What this evaluates

Evaluates multimodal emotion recognition and sentiment analysis capabilities across unimodal and fused modalities, alongside emotional speaker style captioning. It probes how well models capture discrete emotions, continuous sentiment, and fine-grained speaking styles from Chinese dyadic dialogues.

Datasets

• EmotionTalk — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/NKU-HLT/EmotionTalk

Metrics

• ACC (primary) — range: [0, 1]
  • Standard classification accuracy: the proportion of correctly predicted labels out of the total number of samples. Applied to both binary continuous sentiment and discrete emotion classification.
• BLEU_4 — range: [0, 1]
  • Standard BLEU metric using 4-gram overlap between generated captions and reference texts, with brevity penalty.
• ROUGE_L — range: [0, 1]
  • Recall-Oriented Understudy for Gisting Evaluation based on the longest common subsequence between prediction and reference.
• METEOR — range: [0, 1]
  • Metric for evaluation of translation quality that uses synonymy, stemming, and word order alignment.
• SPIDER — range: [0, 1]
  • Semantic and Pragmatic Evaluation metric for dialogue/captioning that measures semantic similarity and discourse coherence.
• FENSE — range: [0, 1]
  • Fine-grained Emotion and Style evaluation metric designed to assess alignment with emotional and stylistic annotations.
• BERTScore — range: [0, 1]
  • Computes token-level similarity using contextual embeddings from BERT, aggregating precision, recall, and F1.
• CLAPScore — range: [0, 1]
  • Measures alignment between generated text and audio features using a Contrastive Language-Audio Pretraining model.

Input / output format

Input: Multimodal inputs (text transcripts, audio recordings, and video frames) representing dyadic conversational turns.

Output: For recognition tasks: discrete emotion labels (4-class or 7-class) or binary sentiment labels (positive/negative). For captioning: natural language text describing speaker, style, emotion, or overall emotional state.

Scoring recipe

def compute_accuracy(preds, gold):
    return sum(p == g for p, g in zip(preds, gold)) / len(gold)

def compute_captioning_metrics(preds, golds):
    # preds and golds are lists of generated/reference strings
    bleu4 = nltk.bleu([gold], pred, weights=(0.25, 0.25, 0.25, 0.25))
    rouge_l = rouge_l_score(gold, pred)
    meteor = meteor_score(gold, pred)
    spider = spider_score(gold, pred)
    fense = fense_score(gold, pred)
    bertscore = bert_score(gold, pred)
    clapscore = clap_score(gold, pred)
    return bleu4, rouge_l, meteor, spider, fense, bertscore, clapscore

Common pitfalls

• Models trained with unimodal labels often outperform those trained with multimodal labels, but this does not necessarily reflect the speaker's true emotional state due to annotation bias.
• Complex fusion algorithms are prone to overfitting on this dataset; simpler strategies like LMF or attention often yield better average performance.
• Frame-level fusion generally underperforms utterance-level fusion for holistic emotion capture.

Evidence (verbatim from paper)

Specifically, in the continuous setting, we focus on a binary classification task that distinguishes between positive and negative emotions, where samples with scores below 0 are labeled as negative, and those above 0 as positive. For the first three tasks, accuracy (ACC) is used as the primary evaluation metric, while for the speaker emotion-style captioning task, we adopt BLEU_4, ROUGE_L, METEOR, SPIDER, FENSE, BERTScore and CLAPScore for evaluation.

Citation

@misc{sun2025emotiontalk,
  title={EmotionTalk: An Interactive Chinese Multimodal Emotion Dataset With Rich Annotations},
  author={Sun et al. (2025)},
  year={2025},
  note={arXiv:2505.23018}
}

• arXiv: 2505.23018