Usc Asr Eval

usc-asr-eval

USC: An Open-Source Uzbek Speech Corpus and Initial Speech Recognition Experiments — Musaev et al. (2021) (arXiv:2107.14419, 2021)

What this evaluates

Evaluates automatic speech recognition (ASR) systems on Uzbek language audio by measuring character and word error rates against manually transcribed ground truth. It probes the model's ability to accurately transcribe low-resource speech data without relying on external linguistic resources or pronunciation dictionaries.

Datasets

• USC — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/IS2AI/Uzbek_ASR

Metrics

• WER (primary) — range: percent
  • Standard edit distance metric: (Substitutions + Deletions + Insertions) / Total words in reference. Expressed as a percentage.
• CER — range: percent
  • Standard edit distance metric: (Substitutions + Deletions + Insertions) / Total characters in reference. Expressed as a percentage.

Input / output format

Input: Audio recordings processed into acoustic features (MFCCs for DNN-HMM; 80-dim filterbank + pitch for E2E), paired with ground truth transcriptions.

Output: Predicted text sequence (character-level for E2E models, word-level for DNN-HMM).

Scoring recipe

def compute_wer_cer(predictions, references):
    total_errors = 0
    total_units = 0
    for pred, ref in zip(predictions, references):
        dist = levenshtein_distance(pred, ref)
        total_errors += dist
        total_units += len(ref)
    return (total_errors / total_units) * 100

Common pitfalls

• Models are evaluated without external linguistic resources (lexicons, pronunciation models), which may inflate error rates compared to resource-rich language benchmarks.
• E2E models output character-level sequences, so WER requires post-hoc word segmentation or graphemic mapping, introducing tokenization ambiguities.
• Performance varies significantly based on language model integration and data augmentation (speed/spectral), making direct model comparisons sensitive to pipeline choices.

Evidence (verbatim from paper)

We conducted speech recognition experiments to demonstrate the reliability of the USC dataset. We built both DNN-HMM and E2E speech recognition models using our dataset (see Section[3]) and evaluated them using the character error rate (CER) and word error rate (WER) metrics. We did not use any external data and other available linguistic resources such as lexicon, pronunciation models, and vocabulary.

Citation

@misc{musaev2021usc,
  title={USC: An Open-Source Uzbek Speech Corpus and Initial Speech Recognition Experiments},
  author={Musaev et al. (2021)},
  year={2021},
  note={arXiv:2107.14419}
}

• arXiv: 2107.14419