Edacc Eval

edacc-eval

The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASR — Sanabria et al. (2023) (arXiv:2303.18110, 2023)

What this evaluates

Evaluates automatic speech recognition (ASR) models on naturalistic, conversational English speech with diverse international accents. It probes the robustness of state-of-the-art ASR systems to real-world speaking conditions and accent variation compared to read-speech benchmarks like LibriSpeech.

Datasets

• EdAcc — total ?; splits: dev (-1), test (-1)

Metrics

• WER (primary) — range: percent
  • Word Error Rate computed at the conversation level. Standard formula: (Substitutions + Deletions + Insertions) / Total Reference Words.

Input / output format

Input: Audio recordings of dyadic video call conversations, decoded in 30-second chunks.

Output: Transcribed text for each audio chunk/conversation.

Scoring recipe

def compute_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        dist = levenshtein_distance(pred.split(), ref.split())
        total_errors += dist
        total_words += len(ref.split())
    return (total_errors / total_words) * 100 if total_words > 0 else 0.0

Common pitfalls

• WER is aggregated at the conversation level rather than per utterance, which can skew averages if conversation lengths vary significantly.
• Accent labels are self-reported by speakers and not verified ground truth, making it difficult to establish a definitive 'true' label for evaluation.
• Models decode audio in fixed 30-second segments, which may introduce boundary artifacts not present in end-to-end streaming evaluation.

Evidence (verbatim from paper)

We start by measuring the general complexity of EdAcc by computing WER on development, and test sets at the conversation level. Because we can only compute one WER for each conversation, we limit this analysis to conversations where both speakers use the same English variety.

Citation

@misc{sanabria2023edacc,
  title={The Edinburgh International Accents of English Corpus: Towards the Democratization of English ASR},
  author={Sanabria et al. (2023)},
  year={2023},
  note={arXiv:2303.18110}
}

• arXiv: 2303.18110