Cameo Eval

cameo-eval

CAMEO: Collection of Multilingual Emotional Speech Corpora — Christop et al. (2025) (arXiv:2505.11051, 2025)

What this evaluates

Evaluates speech emotion recognition (SER) models across multiple languages and emotional states. It probes a model's ability to map raw audio inputs to discrete emotional categories without relying on speaker or language metadata.

Datasets

• CAMEO — total 41265; splits: test (-1)

Metrics

• macro-averaged F1 score (primary) — range: [0, 1]
  • Computes the unweighted mean of the F1 score across all emotion classes, treating each class equally regardless of support.
• weighted F1 score — range: [0, 1]
  • Computes the F1 score for each class and averages them, weighting each class by its support (number of true instances).
• accuracy — range: [0, 1]
  • The proportion of correctly predicted emotional labels out of the total number of instances.

Input / output format

Input: A textual instruction prompt followed by a single audio file. No additional metadata (speaker identity, gender, or language) is provided during inference.

Output: A single-word string corresponding to the predicted emotional state (e.g., 'anger', 'happiness').

Scoring recipe

def post_process_and_score(pred, gold_labels, threshold=0.57):
    words = pred.lower().split()
    label_scores = {lbl: 0.0 for lbl in gold_labels}
    for word in words:
        for lbl in gold_labels:
            ratio = levenshtein_ratio(word, lbl)
            if ratio >= threshold:
                label_scores[lbl] += ratio
    predicted_label = max(label_scores, key=label_scores.get)
    return 1.0 if predicted_label == gold_labels[0] else 0.0
# Aggregate per-class TP/FP/FN to compute macro/weighted F1 and accuracy

Common pitfalls

• Models frequently output adjectives (e.g., 'angry') instead of the required noun labels (e.g., 'anger'), requiring fuzzy matching rather than exact string comparison.
• Generative models often produce multi-word descriptive responses instead of single-word predictions, necessitating word-splitting and score aggregation.
• The 0.57 Levenshtein threshold is specifically calibrated to noun-adjective pairs for the seven primary emotions and may not generalize to other label formats or languages.

Evidence (verbatim from paper)

To capture the performance of the models across different emotions and languages, the evaluation employs widely used metrics, such as macro-averaged F1 score, weighted F1 score and accuracy. ... If a generated response is not an exact match for any of the labels, it is normalized and split into words. Then, the Levenshtein ratio between each target label and each word in the generated response is calculated. Similarity scores below a predefined threshold of 0.57 are filtered out for each label. ... The label with the highest aggregated similarity score is selected as the best match.

Citation

@misc{christop2025cameo,
  title={CAMEO: Collection of Multilingual Emotional Speech Corpora},
  author={Christop et al. (2025)},
  year={2025},
  note={arXiv:2505.11051}
}

• arXiv: 2505.11051