Crowdspeech Eval

crowdspeech-eval

CrowdSpeech and VoxDIY: Benchmark Datasets for Crowdsourced Audio Transcription — Pavlichenko et al. (2021) (arXiv:2107.01091, 2021)

What this evaluates

Evaluates algorithms for aggregating multiple noisy, crowdsourced transcriptions of the same audio recording into a single high-quality reference. It probes how well methods handle sequential textual noise, estimate worker reliability, and adapt across different audio quality domains.

Datasets

• CROWDSPEECH — total ?; splits: dev-clean (-1), dev-other (-1), test-clean (-1), test-other (-1); repo https://github.com/Toloka/CrowdSpeech
• VOXDIY — total ?; splits: RU (-1)
• CROWDWSA2019 — total ?; splits: J1 (-1), T1 (-1), T2 (-1)

Metrics

• WER (primary) — range: percent
  • Average Word Error Rate computed as the normalized edit distance between the aggregated prediction and the ground truth reference transcription, averaged across all recordings in a split. Lower values indicate better performance.

Input / output format

Input: A set of multiple crowdsourced text transcriptions (sequences) for a single audio recording.

Output: A single aggregated text transcription for that recording.

Scoring recipe

def compute_wer(predictions, references):
    wer_scores = []
    for pred, ref in zip(predictions, references):
        score = edit_distance(pred, ref) / max(len(ref.split()), 1)
        wer_scores.append(score)
    return sum(wer_scores) / len(wer_scores) * 100

Common pitfalls

• Using WER on subjective tasks like machine translation (e.g., CROWDWSA2019) is inappropriate because there is no unique ground truth transcription.
• The Oracle baseline assumes access to ground truth to select the best worker annotation, which is unrealistic for deployment but serves only as an upper-bound estimate.
• Ignoring worker reliability/expertise when aggregating sequences can limit performance, as shown by the gap between simple voting methods and more advanced reliability-aware models.

Evidence (verbatim from paper)

Evaluation criterion is the average word error rate (WER) and lower values are better. To evaluate baseline methods, we run them on each of the datasets under consideration excluding the train set of CROWDSPEECH as its main purpose is model training. We then compute the mean value of WER (Word Error Rate) over all recordings in each dataset and report it in Table 4.

Citation

@misc{pavlichenko2021crowdspeech,
  title={CrowdSpeech and VoxDIY: Benchmark Datasets for Crowdsourced Audio Transcription},
  author={Pavlichenko et al. (2021)},
  year={2021},
  note={arXiv:2107.01091}
}

• arXiv: 2107.01091