Peoples Speech Eval

peoples-speech-eval

The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage — Galvez et al. (2021) (arXiv:2111.09344, 2021)

What this evaluates

Evaluates the quality and generalization capability of a large-scale, commercially licensed speech recognition dataset by training an acoustic model on it and measuring word error rate on standard read-speech benchmarks.

Datasets

• The People's Speech — total ?; splits: train (20000)
• Librispeech — total ?; splits: dev-clean (-1), dev-other (-1), test-clean (-1), test-other (-1)

Metrics

• Word Error Rate (WER) (primary) — range: percent
  • Percentage of incorrectly recognized words relative to the total number of words in the reference transcript. Calculated as (Substitutions + Deletions + Insertions) / Total Reference Words.

Input / output format

Input: Audio waveform paired with a reference transcript (normalized to lowercase English alphabet, space character, and apostrophe only).

Output: Predicted transcript generated by a Conformer-CTC model using beam search decoding (beamwidth 1024, alpha 1.0, beta 1.0) with a pre-trained 3-gram language model.

Scoring recipe

def compute_wer(predictions, references):
    # Normalize text: lowercase, keep only a-z, space, apostrophe
    pred = re.sub(r'[^a-z ']', '', predictions.lower())
    ref = re.sub(r'[^a-z ']', '', references.lower())
    # Compute edit distance operations
    subs, dels, ins = edit_distance_operations(pred.split(), ref.split())
    total_words = len(ref.split())
    if total_words == 0: return 0.0
    return (subs + dels + ins) / total_words * 100

Common pitfalls

• The evaluation uses an external benchmark (Librispeech) rather than a held-out split from the dataset itself, so results reflect generalization to read speech, not the dataset's internal conversational/noisy diversity.
• Text normalization strips punctuation and numbers, which may artificially lower WER compared to standard ASR benchmarks that retain them.
• The training subset is filtered by a ≤20% CER threshold, meaning the reported WER reflects performance on a pre-cleaned subset, not the full 30k-hour corpus.

Evidence (verbatim from paper)

Because we chose not to create a test or dev set for this corpus, we sought to evaluate the performance of an acoustic model trained on the dataset on a test set that lacked overlap with The People’s Speech. For this, we chose Librispeech’s test and dev splits. We normalized the text of both The People’s Speech and Librispeech to use only the lowercase English alphabet, space character and apostrophe. | Model and Word Error Rates | dev-clean | dev-other | test-clean | test-other | | --- | --- | --- | --- | --- | | Conformer CTC Model | 9.93% | 25.53% | 9.98% | 26.91% |

Citation

@misc{galvez2021peoplesspeech,
  title={The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage},
  author={Galvez et al. (2021)},
  year={2021},
  note={arXiv:2111.09344}
}

• arXiv: 2111.09344