Taigi Asr Eval

taigi-asr-eval

Breeze Taigi: Benchmarks and Models for Taiwanese Hokkien Speech Recognition and Synthesis — Lan et al. (2026) (arXiv:2603.19259, 2026)

What this evaluates

Evaluates the ability of automatic speech recognition (ASR) systems to transcribe Taiwanese Hokkien (Taigi) audio into text. It specifically measures how well models map Taigi phonetic patterns to Mandarin character sequences using a standardized set of public service announcement recordings.

Datasets

• Taigi ASR Benchmark — total 30; splits: test (30)

Metrics

• Character Error Rate (CER) (primary) — range: percent
  • Ratio of insertions, deletions, and substitutions to the total number of characters in the reference transcription. Computed as (I + D + S) / N, where N is the reference length. Lower values indicate better performance.

Input / output format

Input: Taigi audio file (~30 seconds duration)

Output: Mandarin text transcription

Scoring recipe

def compute_cer(pred: str, ref: str) -> float:
    pred_norm = normalize_text(pred)  # unify numbers, punctuation, case
    ref_norm = normalize_text(ref)
    edits = levenshtein_distance(pred_norm, ref_norm)
    cer = edits / len(ref_norm) if len(ref_norm) > 0 else 0.0
    return cer * 100  # percentage

Common pitfalls

• Failing to normalize outputs for numbers (Arabic vs. Han characters), punctuation, and case before computing CER.
• Interpreting absolute CER values as direct Taigi transcription accuracy, since the ground truth is in Mandarin and the Taigi-to-Mandarin mapping is not strictly one-to-one.
• Comparing systems that output Taigi orthography directly against the Mandarin ground truth without first translating the output.

Evidence (verbatim from paper)

We adopt CER as the primary metric, defined as the ratio of insertions, deletions, and substitutions to the total number of characters in the reference transcription. CER is more appropriate than Word Error Rate (WER) for Han character-based text because Chinese writing does not contain natural word boundaries (spaces between words).

Citation

@misc{lan2026breezetaigi,
  title={Breeze Taigi: Benchmarks and Models for Taiwanese Hokkien Speech Recognition and Synthesis},
  author={Lan et al. (2026)},
  year={2026},
  note={arXiv:2603.19259}
}

• arXiv: 2603.19259