qhjqhj00 tamil-kannada-asr-subword-eval
Subword Dictionary Learning and Segmentation Techniques for Automatic Speech Recognition in Tamil and Kannada — Madhavaraj et al. (2022) (arXiv:2207.13331, 2022)
What this evaluates
Evaluates automatic speech recognition (ASR) systems on agglutinative languages (Tamil and Kannada) to measure how subword dictionary learning and segmentation techniques (Morfessor, BPE, extended-BPE) reduce out-of-vocabulary rates and improve word error rates compared to baseline word-level models.
Datasets
- Tamil and Kannada ASR dataset — total ?; splits: test (-1)
Metrics
WER(primary) — range: percent- Word Error Rate (WER) measures the percentage of words incorrectly recognized in a transcript relative to the ground truth reference. Calculated as (S + D + I) / N, where S=substitutions, D=deletions, I=insertions, and N=total words in the reference transcript.
OOV rate— range: percent- Out-of-Vocabulary (OOV) rate is defined as the ratio of the number of words in the test data that are not present in the training corpus to the total number of words in the test data.
Input / output format
Input: Audio speech recordings in Tamil or Kannada.
Output: Transcribed text sequence corresponding to the audio input.
Scoring recipe
def compute_wer(hypothesis, reference):
edits = levenshtein_distance(hypothesis.split(), reference.split())
return (edits / len(reference.split())) * 100
def compute_oov_rate(test_transcript, training_vocab):
test_words = test_transcript.split()
oov_count = sum(1 for w in test_words if w not in training_vocab)
return (oov_count / len(test_words)) * 100
Common pitfalls
- OOV rate is calculated against the training vocabulary, not the subword dictionary size, so a 0% OOV rate does not imply perfect ASR accuracy.
- WER improvements are highly dependent on the subword n-gram language model order (3-gram to 6-gram); comparing across different LM orders without normalization is misleading.
- ML and Viterbi segmentation techniques differ in how they estimate subword parameters (soft-weighing all paths vs. best path), which directly impacts WER and should not be conflated with dictionary creation method effects.
Evidence (verbatim from paper)
OOV rate is defined as the ratio of number of words in the test data which are not present in the training corpus to the total number of words in the test data. We compare the performances of subword-ASRs with the baseline ASR system in terms of OOV rate and WER and empirically justify the need for subword modeling to handle highly agglutinative languages like Tamil and Kannada.
Citation
@misc{madhavaraj2022subword,
title={Subword Dictionary Learning and Segmentation Techniques for Automatic Speech Recognition in Tamil and Kannada},
author={Madhavaraj et al. (2022)},
year={2022},
note={arXiv:2207.13331}
}
- arXiv: 2207.13331