Faetar Benchmark Eval

faetar-benchmark-eval

The Faetar Benchmark: Speech Recognition in a Very Under-Resourced Language — Ong et al. (2024) (arXiv:2409.08103, 2024)

What this evaluates

Evaluates automatic speech recognition (ASR) models on a highly under-resourced language (Faetar/Franco-Provençal) characterized by noisy field recordings, lack of standard orthography, and inconsistent phonetic transcriptions.

Datasets

• Faetar Benchmark — total ?; splits: test (-1)

Metrics

• PER (primary) — range: percent
  • Phone Error Rate calculated as the total number of alignment errors (substitutions, insertions, deletions) between the reference and hypothesis transcripts divided by the total number of phones in the reference transcript, multiplied by 100. Alignment uses the unit-cost Levenshtein algorithm.

Input / output format

Input: Audio recordings of Faetar speech utterances

Output: Phonetic hypothesis transcript

Scoring recipe

total_errors = 0
total_ref_phones = 0
for u in test_set:
    ref = get_reference_phones(u)
    hyp = get_hypothesis_phones(u)
    errors = levenshtein_distance(ref, hyp, unit_cost=True)
    total_errors += errors
    total_ref_phones += len(ref)
per = (total_errors / total_ref_phones) * 100

Common pitfalls

• Using Word Error Rate (WER) instead of PER, which is too severe due to substantial phonetic variation in the transcriptions.
• Using Character Error Rate (CER) without accounting for how spaces, affricates, and long phones are treated, which can cause minor but notable deviations from PER.

Evidence (verbatim from paper)

The measure of performance is phone error rate (PER) on the test set, calculated by aligning reference and hypothesis transcripts by the unit-cost Levenshtein algorithm. Given all utterances $u$ in a partition $\mathcal{U}$, the number of phones in a reference transcript $n_{u}$, and the number of errors in a model’s hypothesis transcript $e_{u}$, the PER on $\mathcal{U}$ is defined as: $\text{PER}{\mathcal{U}}=\frac{\sum{u\in\mathcal{U}}e_{u}}{\sum_{u\in\mathcal{U}}n_{u}}\times 100%.$

Citation

@misc{ong2024faetar,
  title={The Faetar Benchmark: Speech Recognition in a Very Under-Resourced Language},
  author={Ong et al. (2024)},
  year={2024},
  note={arXiv:2409.08103}
}

• arXiv: 2409.08103