Ser Multilingual Probing Eval

ser-multilingual-probing-eval

Decoding Emotions: A comprehensive Multilingual Study of Speech Models for Speech Emotion Recognition — Anant Singh et al. (2023) (arXiv:2308.08713, 2023)

What this evaluates

Evaluates pre-trained speech models' ability to recognize emotions in audio across multiple languages. It specifically tests how internal layer representations and feature aggregation strategies impact classification performance.

Datasets

• AESDD — total ?; splits: test (-1)
• CaFE — total ?; splits: test (-1)
• EmoDB — total ?; splits: test (-1)
• EMOVO — total ?; splits: test (-1)
• IEMOCAP — total ?; splits: test (-1)
• RAVDESS — total ?; splits: test (-1)
• ShEMO — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • The proportion of correctly predicted emotion labels out of the total number of instances. Calculated as (number of correct predictions) / (total predictions).

Input / output format

Input: Raw speech audio files.

Output: Discrete emotion category label.

Scoring recipe

correct = sum(1 for p, g in zip(predictions, gold) if p == g)
total = len(gold)
return (correct / total) * 100

Common pitfalls

• Relying on final transformer layers or aggregated multi-layer features instead of center layers, which contain the richest contextual features for emotion recognition.
• Using ASR-finetuned models, which discard prosodic information crucial for emotion recognition and consistently underperform non-ASR counterparts.
• Evaluating on a single run without averaging over multiple trials, as the protocol requires each trial to be run five times for statistical validity.

Evidence (verbatim from paper)

The exact values of classification accuracies for the dense probing head with the corresponding best layer for each model are shown in Table 3.

Citation

@misc{singh2023decodingemotions,
  title={Decoding Emotions: A comprehensive Multilingual Study of Speech Models for Speech Emotion Recognition},
  author={Anant Singh et al. (2023)},
  year={2023},
  note={arXiv:2308.08713}
}

• arXiv: 2308.08713