# Emotiontalk Eval

> Evaluates multimodal emotion recognition and sentiment analysis capabilities across unimodal and fused modalities, alongside emotional speaker style captioning. It probes how well models capture discrete emotions, continuous sentiment, and fine-grained speaking styles from Chinese dyadic dialogues. Use when the user wants to benchmark on EmotionTalk, or asks about evaluating this task. Reports ACC.

- Skill: `qhjqhj00/emotiontalk-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/emotiontalk-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/emotiontalk-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/emotiontalk-eval

---


# emotiontalk-eval

> EmotionTalk: An Interactive Chinese Multimodal Emotion Dataset With Rich Annotations — Sun et al. (2025) (arXiv:2505.23018, 2025)

## What this evaluates

Evaluates multimodal emotion recognition and sentiment analysis capabilities across unimodal and fused modalities, alongside emotional speaker style captioning. It probes how well models capture discrete emotions, continuous sentiment, and fine-grained speaking styles from Chinese dyadic dialogues.

## Datasets

- **EmotionTalk** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/NKU-HLT/EmotionTalk

## Metrics

- `ACC` **(primary)** — range: [0, 1]
  - Standard classification accuracy: the proportion of correctly predicted labels out of the total number of samples. Applied to both binary continuous sentiment and discrete emotion classification.
- `BLEU_4` — range: [0, 1]
  - Standard BLEU metric using 4-gram overlap between generated captions and reference texts, with brevity penalty.
- `ROUGE_L` — range: [0, 1]
  - Recall-Oriented Understudy for Gisting Evaluation based on the longest common subsequence between prediction and reference.
- `METEOR` — range: [0, 1]
  - Metric for evaluation of translation quality that uses synonymy, stemming, and word order alignment.
- `SPIDER` — range: [0, 1]
  - Semantic and Pragmatic Evaluation metric for dialogue/captioning that measures semantic similarity and discourse coherence.
- `FENSE` — range: [0, 1]
  - Fine-grained Emotion and Style evaluation metric designed to assess alignment with emotional and stylistic annotations.
- `BERTScore` — range: [0, 1]
  - Computes token-level similarity using contextual embeddings from BERT, aggregating precision, recall, and F1.
- `CLAPScore` — range: [0, 1]
  - Measures alignment between generated text and audio features using a Contrastive Language-Audio Pretraining model.

## Input / output format

**Input**: Multimodal inputs (text transcripts, audio recordings, and video frames) representing dyadic conversational turns.

**Output**: For recognition tasks: discrete emotion labels (4-class or 7-class) or binary sentiment labels (positive/negative). For captioning: natural language text describing speaker, style, emotion, or overall emotional state.

## Scoring recipe

```python
def compute_accuracy(preds, gold):
    return sum(p == g for p, g in zip(preds, gold)) / len(gold)

def compute_captioning_metrics(preds, golds):
    # preds and golds are lists of generated/reference strings
    bleu4 = nltk.bleu([gold], pred, weights=(0.25, 0.25, 0.25, 0.25))
    rouge_l = rouge_l_score(gold, pred)
    meteor = meteor_score(gold, pred)
    spider = spider_score(gold, pred)
    fense = fense_score(gold, pred)
    bertscore = bert_score(gold, pred)
    clapscore = clap_score(gold, pred)
    return bleu4, rouge_l, meteor, spider, fense, bertscore, clapscore
```

## Common pitfalls

- Models trained with unimodal labels often outperform those trained with multimodal labels, but this does not necessarily reflect the speaker's true emotional state due to annotation bias.
- Complex fusion algorithms are prone to overfitting on this dataset; simpler strategies like LMF or attention often yield better average performance.
- Frame-level fusion generally underperforms utterance-level fusion for holistic emotion capture.

## Evidence (verbatim from paper)

> Specifically, in the continuous setting, we focus on a binary classification task that distinguishes between positive and negative emotions, where samples with scores below 0 are labeled as negative, and those above 0 as positive. For the first three tasks, accuracy (ACC) is used as the primary evaluation metric, while for the speaker emotion-style captioning task, we adopt BLEU_4, ROUGE_L, METEOR, SPIDER, FENSE, BERTScore and CLAPScore for evaluation.

## Citation

```bibtex
@misc{sun2025emotiontalk,
  title={EmotionTalk: An Interactive Chinese Multimodal Emotion Dataset With Rich Annotations},
  author={Sun et al. (2025)},
  year={2025},
  note={arXiv:2505.23018}
}
```

- arXiv: 2505.23018