Phoneme Level Asr Eval

phoneme-level-asr-eval

Hard to Be Heard: Phoneme-Level ASR Analysis of Phonologically Complex, Low-Resource Endangered Languages — Akavarapu et al. (2026) (arXiv:2604.18204, 2026)

What this evaluates

Evaluates automatic speech recognition performance on two low-resource, phonologically complex endangered languages (Archi and Kina Rutul) at the word, character, and phoneme levels. It specifically probes how training data frequency impacts phoneme recognition accuracy and error types.

Datasets

• Archi & Kina Rutul ASR — total ?; splits: train (-1), val (-1), test (-1); HF mahesh27/archi_rutul_asr; repo https://github.com/mahesh-ak/north_caucasian_asr

Metrics

• PER (primary) — range: [0, 1]
  • Normalized edit distance at the phoneme level. Precision, recall, and F1 are derived from true positives (N), substitutions (S), insertions (I), and deletions (D): pr = N/(N+S+I), re = N/(N+S+D), F1 = 2·pr·re/(pr+re).
• WER — range: [0, 1]
  • Normalized edit distance at the word level.
• CER — range: [0, 1]
  • Normalized edit distance at the character level.
• Phoneme F1 — range: [0, 1]
  • Harmonic mean of phoneme-level precision and recall: F1 = 2·pr·re/(pr+re).

Input / output format

Input: Raw audio recordings of speech in Archi or Kina Rutul.

Output: Transcribed text in either IPA or Cyrillic script, without translation, interpretation, or punctuation.

Scoring recipe

def compute_asr_metrics(predictions, references):
    wer = normalized_edit_distance(predictions, references, level='word')
    cer = normalized_edit_distance(predictions, references, level='char')
    per = normalized_edit_distance(predictions, references, level='phoneme')
    N, S, I, D = count_phoneme_edits(predictions, references)
    pr = N / (N + S + I)
    re = N / (N + S + D)
    f1 = 2 * pr * re / (pr + re)
    return {'WER': wer, 'CER': cer, 'PER': per, 'pr': pr, 're': re, 'F1': f1}

Common pitfalls

• Confusing phoneme-level error rates with word/character rates, especially given the complex phonotactics and non-standard orthographies of these languages.
• Assuming low PER stems from model architecture limitations rather than data scarcity, as performance strongly correlates with training frequency via a sigmoid curve.
• Using incorrect CTC decoding parameters; the paper specifies a 3-gram LM with alpha=beta=0.3 and a fixed beam size of 10.

Evidence (verbatim from paper)

We evaluate on standard metrics for ASR — word, character and phoneme error rates, respectively WER, CER and PER which are normalized edit distances respectively at levels of words, characters and phonemes. We further store the number of edits — insertions (I), deletions (D) and substitutions (S) — along with true positives (N) for each phoneme to compute phoneme-level precision (pr), recall (re) and F1 scores:

Citation

@misc{akavarapu2026hard,
  title={Hard to Be Heard: Phoneme-Level ASR Analysis of Phonologically Complex, Low-Resource Endangered Languages},
  author={Akavarapu et al. (2026)},
  year={2026},
  note={arXiv:2604.18204}
}

• arXiv: 2604.18204