Unseen Speaker Ser Eval

unseen-speaker-ser-eval

Exploring Multilingual Unseen Speaker Emotion Recognition: Leveraging Co-Attention Cues in Multitask Learning — Goel et al. (2024) (arXiv:2406.08931, 2024)

What this evaluates

Evaluates a model's ability to recognize emotions in speech from speakers it has never encountered during training. It probes cross-speaker generalization and robustness to acoustic variability across multiple languages and recording conditions.

Datasets

• CREMA-D — total ?; splits: train (-1), val (-1), test (-1)
• IEMOCAP — total ?; splits: train (-1), val (-1), test (-1)
• RAVDESS — total ?; splits: train (-1), val (-1), test (-1)
• EmoDB — total ?; splits: train (-1), val (-1), test (-1)
• CaFE — total ?; splits: train (-1), val (-1), test (-1)
• BhavVani — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• WF1 (primary) — range: [0, 1]
  • Weighted F1 score: harmonic mean of precision and recall per class, averaged across all classes weighted by their support (number of true instances per class). Accounts for class imbalance.
• WA — range: [0, 1]
  • Weighted accuracy: proportion of correctly predicted samples, weighted by class support. Accounts for class imbalance.

Input / output format

Input: Raw audio waveform, optionally processed into MFCCs (40 coefficients, hop length 160) and spectrograms (Hamming window, length 40, hop 10, FFT 800), or 1D embeddings from pre-trained encoders (Wav2Vec2.0, WavLM, HuBERT, Whisper) via average pooling of 2D time-frequency outputs.

Output: Categorical emotion label

Scoring recipe

def compute_metrics(y_true, y_pred, classes):
    correct = sum(1 for t, p in zip(y_true, y_pred) if t == p)
    wa = correct / len(y_true)
    f1_scores = []
    for c in classes:
        tp = sum(1 for t, p in zip(y_true, y_pred) if t == c and p == c)
        fp = sum(1 for t, p in zip(y_true, y_pred) if t != c and p == c)
        fn = sum(1 for t, p in zip(y_true, y_pred) if t == c and p != c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        support = sum(1 for t in y_true if t == c)
        f1_scores.append(f1 * support)
    wf1 = sum(f1_scores) / len(y_true)
    return wa, wf1

Common pitfalls

• Using speaker-dependent splits instead of leave-speaker-out, which leaks acoustic identity and artificially inflates performance.
• Reporting standard accuracy instead of weighted metrics, which masks poor performance on minority emotion classes due to dataset imbalance.
• Averaging metrics per fold instead of pooling predictions across all 10 folds before computing the final score, leading to incorrect weighting.

Evidence (verbatim from paper)

For evaluating performance on unseen speakers, we followed 10-fold leave-speaker-out cross validation, wherein each dataset was segmented into 10 folds with each fold containing unique speakers... Note: WA stands for weighted accuracy and WF1 stands for weighted F1 score. These measures account for class imbalance.

Citation

@misc{goel2024camu,
  title={Exploring Multilingual Unseen Speaker Emotion Recognition: Leveraging Co-Attention Cues in Multitask Learning},
  author={Goel et al. (2024)},
  year={2024},
  note={arXiv:2406.08931}
}

• arXiv: 2406.08931