Emogator Eval

emogator-eval

EmoGator: A New Open Source Vocal Burst Dataset with Baseline Machine Learning Classification Methodologies — Buhl (2023) (arXiv:2301.00508, 2023)

What this evaluates

Evaluates machine learning models' ability to classify brief, nonverbal vocal bursts into discrete emotional states. It probes the limits of audio classification on highly ambiguous, short-duration human vocalizations across 30 affective categories.

Datasets

• EmoGator — total 32130; splits: train (-1), val (-1), test (-1); repo https://github.com/fredbuhl/EmoGator

Metrics

• F1 score (primary) — range: [0, 1]
  • The unweighted mean of per-class F1 scores. Calculated as the average of precision and recall F1 for each of the 30 emotion categories, treating all classes equally regardless of support.

Input / output format

Input: Raw audio waveform files representing short vocal bursts.

Output: A single predicted emotion category label from a predefined set of 30 classes.

Scoring recipe

def f1_score(preds, gold, classes):
    f1s = []
    for c in classes:
        tp = sum(1 for p, g in zip(preds, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
    return sum(f1s) / len(f1s)

Common pitfalls

• Data augmentation via independent pitch and tempo shifts consistently degraded model performance compared to the original training set.
• Preprocessing steps like denoising or trimming silence from audio endpoints reduced classification accuracy.
• Evaluating on reduced category subsets (e.g., 10 or 16 classes) artificially inflates F1 scores and does not reflect full-dataset capability.

Evidence (verbatim from paper)

For one-dimensional convolutional neural networks, the best results against the full dataset were with a 70% / 15% / 15% train/validation/test split, using an 18-layer 1D CNN based on [[31]], but with dropout layers after each convolution. For the full 30-category dataset, the average F1 score was 0.270.

Citation

@misc{buhl2023emogator,
  title={EmoGator: A New Open Source Vocal Burst Dataset with Baseline Machine Learning Classification Methodologies},
  author={Buhl (2023)},
  year={2023},
  note={arXiv:2301.00508}
}

• arXiv: 2301.00508