Ser Multiwindow Eval

ser-multiwindow-eval

Multi-Window Data Augmentation Approach for Speech Emotion Recognition — Padi et al. (2020) (arXiv:2010.09895, 2020)

What this evaluates

Evaluates deep learning models for speech emotion recognition (SER) using a multi-window data augmentation strategy. It probes the model's ability to classify categorical emotions from speech audio under varying feature extraction window sizes and class configurations.

Datasets

• IEMOCAP — total 4490; splits: train (-1), test (-1)
• RAVDESS — total ?; splits: train (-1), test (-1)
• SAVEE — total ?; splits: train (-1), test (-1)

Metrics

• Unweighted Accuracy (UA) (primary) — range: percent
  • Macro average of per-class accuracies, calculated as the mean of accuracy scores for each emotion class regardless of class frequency.
• Weighted Average Precision (WAP) — range: percent
  • Weighted average of per-class precision scores, where weights correspond to the support (number of true instances) for each class.
• Weighted Average F1 (WAF1) — range: percent
  • Weighted average of per-class F1 scores, calculated as the harmonic mean of precision and recall for each class, averaged using class support as weights.

Input / output format

Input: Raw speech audio files, processed into speech-based features using multiple window sizes (25ms, 50ms, 100ms, 200ms) with 50% overlap.

Output: Categorical emotion label (e.g., 'angry', 'happy', 'neutral', 'sad', 'excited', 'fear', 'disgust').

Scoring recipe

def compute_metrics(y_true, y_pred):
    classes = sorted(set(y_true + y_pred))
    class_accs = [sum(t==p for t,p in zip(y_true, y_pred) if t==c) / max(sum(1 for t in y_true if t==c), 1) for c in classes]
    ua = sum(class_accs) / len(classes)
    precisions = [precision_score(y_true, y_pred, labels=[c], average=None)[0] for c in classes]
    recalls = [recall_score(y_true, y_pred, labels=[c], average=None)[0] for c in classes]
    f1s = [2 * p * r / (p + r) if (p + r) > 0 else 0 for p, r in zip(precisions, recalls)]
    supports = [sum(1 for t in y_true if t == c) for c in classes]
    total_support = sum(supports)
    wap = sum(p * s for p, s in zip(precisions, supports)) / total_support
    waf1 = sum(f * s for f, s in zip(f1s, supports)) / total_support
    return ua, wap, waf1

Common pitfalls

• Do not use cross-validation; the paper explicitly uses a fixed 80/20 train/test split due to computational constraints.
• For RAVDESS, only spoken sentences should be used; sung sentences must be excluded.
• RAVDESS requires merging 'neutral' and 'calm' into a single 'neutral' class.
• IEMOCAP has two distinct class configurations (Exp 1: happy; Exp 2: excited) that must not be mixed.

Evidence (verbatim from paper)

In our evaluations, we use 80% of the data for training and 20% of data for testing purposes because performing cross-validation on deep learning models with varying window sizes is not feasible in terms of time and computational requirements. We present our findings by reporting Unweighted Accuracy (UA), Weighted Average Precision (WAP), and Weighted Average F1 (WAF1) measures.

Citation

@misc{padi2020multiwindow,
  title={Multi-Window Data Augmentation Approach for Speech Emotion Recognition},
  author={Padi et al. (2020)},
  year={2020},
  note={arXiv:2010.09895}
}

• arXiv: 2010.09895