Berst Eval

berst-eval

BERSting at the Screams: A Benchmark for Distanced, Emotional and Shouted Speech Recognition — Tuttösí et al. (2025) (arXiv:2505.00059, 2025)

What this evaluates

Probes the robustness of Automatic Speech Recognition (ASR) and Speech Emotion Recognition (SER) models under challenging real-world conditions, including varying distances, physical obstructions, and high-intensity vocalizations. It specifically tests whether models can maintain accuracy when linguistic context is removed via nonsense phrases and when acoustic features are degraded by far-field recording and shouting.

Datasets

• BERSt — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• ASR performance (primary) — range: [0, 1]
  • Word Error Rate calculated as the minimum number of insertions, deletions, and substitutions required to transform the predicted transcript into the gold transcript, normalized by the length of the gold transcript.
• SER accuracy — range: [0, 1]
  • Classification accuracy measuring the proportion of correctly predicted emotion labels (neutral, anger, joy, fear, disgust, surprise, sadness) out of the total number of test instances.

Input / output format

Input: Raw audio recordings captured via smartphone microphones at varying distances (from next to face to opposite side of room/wall), with physical obstructions (bags, pockets, hands) and surface variations. Each clip contains a random nonsense phrase spoken at one of three intensity levels (speak, shout, scream) and prompted with one of seven basic emotions.

Output: For ASR: the transcribed text of the phrase. For SER: the predicted emotion label (neutral, anger, joy, fear, disgust, surprise, sadness) and shout level (shouting/not-shouting).

Scoring recipe

def compute_asr_wer(predictions, gold_transcripts):
    total_errors = 0
    total_words = 0
    for pred, gold in zip(predictions, gold_transcripts):
        total_errors += levenshtein_distance(pred, gold)
        total_words += len(gold.split())
    return total_errors / total_words if total_words > 0 else 0.0

def compute_ser_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels) if gold_labels else 0.0

Common pitfalls

• The dataset uses deliberately nonsensical phrases to block linguistic context, which can artificially inflate error rates compared to benchmarks using natural speech.
• Distance, obstructions, and shout intensity are highly confounded; models may exploit loudness or acoustic artifacts rather than learning true robustness or emotional cues.
• Emotion ground-truth labels are explicitly noted as unvalidated ('Emotion validation has not been completed and remains future work'), so SER evaluation relies on potentially noisy or subjective prompts.

Evidence (verbatim from paper)

ASR performance drops with distance and shout intensity, while SER shows poor generalization across emotions regardless of distance—highlighting a critical gap in real-world robustness for expressive, far-field speech recognition.

Citation

@misc{tuttos2025berst,
  title={BERSting at the Screams: A Benchmark for Distanced, Emotional and Shouted Speech Recognition},
  author={Tuttösí et al. (2025)},
  year={2025},
  note={arXiv:2505.00059}
}

• arXiv: 2505.00059