Imigue Speech Eval

imigue-speech-eval

iMiGUE-Speech: A Spontaneous Speech Dataset for Affective Analysis — Kakouros et al. (2026) (arXiv:2602.21464, 2026)

What this evaluates

Evaluates models on recognizing spontaneous emotional states from unscripted speech and text. It probes acoustic prosody through dimensional regression and categorical classification, as well as linguistic sentiment polarity in real-world sports interview contexts.

Datasets

• iMiGUE-Speech — total ?; splits: test (-1); repo https://github.com/CV-AC/imigue-speech

Metrics

• Dimensional Emotion Regression (Arousal, Dominance, Valence) — range: other
  • Standard regression metrics (e.g., MAE, Pearson correlation) applied to continuous predictions of physiological activation (Arousal), perceived control (Dominance), and qualitative pleasantness (Valence).
• Categorical Emotion Classification (primary) — range: [0, 1]
  • Standard classification metrics (e.g., accuracy, F1) for mapping speech segments into discrete emotional categories.
• Transcript-based Sentiment Analysis — range: [0, 1]
  • Binary classification accuracy or F1 for mapping transcript segments into Positive or Negative sentiment polarity.

Input / output format

Input: Segmented audio tracks of interviewees (athletes) and their corresponding transcripts.

Output: Continuous values for Arousal, Dominance, and Valence; discrete emotion category labels; or binary sentiment labels (Positive/Negative).

Scoring recipe

# Dimensional Regression (Arousal, Dominance, Valence)
mae = mean(abs(pred_dim - gold_dim))

# Categorical Emotion Classification
accuracy = sum(pred_cat == gold_cat) / len(gold_cat)

# Transcript-based Sentiment Analysis
sentiment_acc = sum(pred_sent == gold_sent) / len(gold_sent)

Common pitfalls

• Evaluations are strictly limited to segmented interviewee (athlete) audio, excluding interviewer turns and full conversational context.
• The paper benchmarks transfer learning using models pre-trained and fine-tuned on external corpora (MSP-Podcast, Odyssey, diverse sentiment datasets) rather than training from scratch on iMiGUE-Speech.
• Spontaneous, unscripted prosody differs significantly from acted or scripted emotion datasets, so standard SER baselines may underperform without domain-specific adaptation.

Evidence (verbatim from paper)

To empirically demonstrate the utility of the iMiGUE-Speech extensions and to establish initial performance benchmarks, we operationalize the application scenarios outlined in Section 3 into two concrete evaluation tasks: Speech Emotion Recognition (SER) and Transcript-based Sentiment Analysis. These tasks leverage state-of-the-art pre-trained representations to assess the dataset’s capacity for capturing spontaneous affective states from both acoustic and linguistic modalities. ... This task involves mapping the speech segments into discrete emotional categories... The objective is to classify each transcript segment into Positive or Negative sentiment polarity.

Citation

@misc{kakouros2026imiguespeech,
  title={iMiGUE-Speech: A Spontaneous Speech Dataset for Affective Analysis},
  author={Kakouros et al. (2026)},
  year={2026},
  note={arXiv:2602.21464}
}

• arXiv: 2602.21464