Kashmiri Tts Eval

kashmiri-tts-eval

Bolbosh: Script-Aware Flow Matching for Kashmiri Text-to-Speech — Tajamul Ashraf et al. (2026) (arXiv:2603.07513, 2026)

What this evaluates

Evaluates the acoustic quality, intelligibility, and diacritic sensitivity of a Kashmiri text-to-speech system. It probes the model's ability to accurately map Perso-Arabic script with explicit diacritics to natural-sounding speech and maintain spectral fidelity under low-resource conditions.

Datasets

• Curated Kashmiri Corpus — total ?; splits: test (-1); repo https://github.com/gaash-lab/Bolbosh

Metrics

• MCD (primary) — range: other
  • Mel-Cepstral Distortion computed by extracting Mel-Generalized Cepstral Coefficients (MCEPs) via the WORLD vocoder (excluding the 0th coefficient), aligning synthesized and reference utterances using Dynamic Time Warping (DTW), and calculating the scaled Euclidean distance along the optimal path. Lower values indicate better spectral similarity.
• MOS (primary) — range: other
  • Mean Opinion Score from a listening study with 32 native Kashmiri speakers rating intelligibility and prosodic naturalness on a 5-point scale (1: unintelligible, 5: perfectly natural and intelligible). Higher values indicate better perceived quality.
• WER — range: percent
  • Word Error Rate computed using a proxy ASR system (IndicConformer RNN-T without diacritics). Treated as supplementary due to high baseline ASR error rates on Kashmiri.
• rWER — range: percent
  • Relative Word Error Rate normalized against the proxy ASR's error rate on ground-truth recordings to isolate synthesis errors from inherent ASR limitations.

Input / output format

Input: Text input using a 272-grapheme vocabulary that explicitly preserves Kashmiri diacritics, paired with reference audio for objective metric computation.

Output: Synthesized audio waveform for each input text utterance.

Scoring recipe

def compute_metrics(reference_audio, synthesized_audio, text):
    # MCD
    ref_mcep = extract_mcep(reference_audio, world_vocoder, exclude_0th=True)
    syn_mcep = extract_mcep(synthesized_audio, world_vocoder, exclude_0th=True)
    alignment = dtw(ref_mcep, syn_mcep)
    mcd = mean(scaled_euclidean_distance(ref_mcep, syn_mcep, alignment))
    
    # MOS
    ratings = [speaker.score(synthesized_audio, scale=1..5) for _ in range(32)]
    mos = mean(ratings)
    
    # WER & rWER
    asr = load_asr('indic-conformer_rnnt_no_diacritics')
    ref_wer = asr.error_rate(reference_audio)
    syn_wer = asr.error_rate(synthesized_audio)
    wer = syn_wer
    rwer = (syn_wer - ref_wer) / ref_wer * 100
    return mcd, mos, wer, rwer

Common pitfalls

• Removing diacritics from input text causes severe performance degradation (e.g., rWER jumps from 4.14% to 13.23%), so explicit diacritic modeling is mandatory for valid evaluation.
• Absolute WER is unreliable due to underdeveloped Kashmiri ASR; rWER must be used to normalize for baseline ASR errors.
• MCD requires DTW alignment and WORLD vocoder MCEP extraction; skipping DTW or using different feature extractors yields non-comparable results.

Evidence (verbatim from paper)

We assess synthesis quality using both objective and subjective measures. Objective fidelity is evaluated with Mel-Cepstral Distortion (MCD) [30]. To account for speaking-rate differences, synthesized and reference utterances are aligned using Dynamic Time Warping (DTW). Mel-Generalized Cepstral Coefficients (MCEPs) are extracted via the WORLD vocoder, excluding the 0th coefficient, and MCD is computed as the scaled Euclidean distance along the optimal alignment path, where lower values indicate greater spectral similarity. Subjective quality is measured through Mean Opinion Score (MOS) [31]. We conducted a listening study with 32 native Kashmiri speakers who rated intelligibility and prosodic naturalness on a 5-point scale (1: unintelligible, 5: perfectly natural and intelligible). We additionally report Word Error Rate (WER) using a proxy ASR system... To separate synthesis errors from inherent ASR limitations, we compute Relative WER (rWER) normalized against ASR performance on ground-truth recordings.

Citation

@misc{ashraf2026bolbosh,
  title={Bolbosh: Script-Aware Flow Matching for Kashmiri Text-to-Speech},
  author={Tajamul Ashraf et al. (2026)},
  year={2026},
  note={arXiv:2603.07513}
}

• arXiv: 2603.07513