Deepdialogue Ser Eval

deepdialogue-ser-eval

DeepDialogue: A Multi-Turn Emotionally-Rich Spoken Dialogue Dataset — Koudounas et al. (2025) (arXiv:2505.19978, 2025)

What this evaluates

Evaluates the emotional expressivity and transferability of a generated multi-turn spoken dialogue dataset by training speech emotion recognition models and measuring their classification performance on held-out and zero-shot test sets.

Datasets

• DeepDialogue (SER subset) — total ?; splits: train (-1), test (-1)
• RAVDESS — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Standard classification accuracy: the proportion of correctly predicted emotion labels out of the total number of instances.
• macro F1-score — range: percent
  • Macro-averaged F1-score: the unweighted mean of the F1-scores computed independently for each emotion category.

Input / output format

Input: Audio recordings of dialogue turns paired with ground-truth emotion labels.

Output: Predicted emotion category label for each audio instance.

Scoring recipe

def compute_metrics(preds, golds, num_classes):
    accuracy = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    f1_scores = []
    for c in range(num_classes):
        tp = sum(1 for p, g in zip(preds, golds) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, golds) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, golds) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    macro_f1 = sum(f1_scores) / num_classes
    return accuracy, macro_f1

Common pitfalls

• The training subset is artificially balanced (1,000 turns per emotion category), which may inflate performance compared to real-world imbalanced dialogue data.
• Zero-shot evaluation on RAVDESS suffers from domain shift, so the reported accuracy drop does not necessarily reflect model failure but rather dataset mismatch.
• The exact number of emotion categories is not explicitly stated in the text, making it difficult to verify the macro F1 calculation or compare with other benchmarks.

Evidence (verbatim from paper)

All models demonstrate strong performance (around 90%) in both accuracy and macro F1-score when evaluated on a held-out test set from DeepDialogue. To assess the generalizability of the learned representations, we evaluate the best-performing model in a zero-shot setting on the RAVDESS dataset, which shares the same emotion label distribution. As expected, we observe a performance drop due to domain shift; however, results remain strong. Our zero-shot HuBERT-DD achieves an accuracy of 56.6%, closely approaching the performance of a linear-probing baseline (HuBERT-LP, 65.3%) trained directly on RAVDESS.

Citation

@misc{koudounas2025deepdialogue,
  title={DeepDialogue: A Multi-Turn Emotionally-Rich Spoken Dialogue Dataset},
  author={Koudounas et al. (2025)},
  year={2025},
  note={arXiv:2505.19978}
}

• arXiv: 2505.19978