Blizzard Indian G2p Dur Eval

blizzard-indian-g2p-dur-eval

DNN-based Speech Synthesis for Indian Languages from ASCII text — Ronanki et al. (2016) (arXiv:1608.05374, 2016)

What this evaluates

Evaluates DNN-based grapheme-to-phoneme and duration prediction models for three Indian languages (Hindi, Tamil, Telugu) using crowdsourced ASCII transliterated text. It measures the accuracy of predicted phoneme durations against forced-aligned ground truth to assess component quality for speech synthesis.

Datasets

• Blizzard Challenge 2015 (Hindi, Tamil, Telugu) — total 5653; splits: train (-1), dev (-1), test (-1)

Metrics

• RMSE (frames per phone) (primary) — range: other
  • Root Mean Square Error between predicted and forced-aligned phoneme durations. Calculated as the square root of the average of squared differences between predicted and ground truth duration values.

Input / output format

Input: ASCII transliterated text, tokenized by whitespace and punctuation.

Output: Predicted phoneme durations in frames.

Scoring recipe

def compute_rmse(predictions, gold):
    if len(predictions) != len(gold):
        raise ValueError('Length mismatch')
    squared_diffs = [(p - g) ** 2 for p, g in zip(predictions, gold)]
    mse = sum(squared_diffs) / len(squared_diffs)
    return math.sqrt(mse)

Common pitfalls

• Splits are stratified by annotator rather than randomly split, meaning different annotators produced the train, dev, and test data to prevent leakage.
• The train/dev/test split ratios are 92%/4%/4%, which deviates from the common 90/5/5 or 80/10/10 conventions and may cause confusion.

Evidence (verbatim from paper)

Table 4: RMSE (frames per phone) between predicted and forced-aligned durations.

Citation

@misc{ronanki2016dnn,
  title={DNN-based Speech Synthesis for Indian Languages from ASCII text},
  author={Ronanki et al. (2016)},
  year={2016},
  note={arXiv:1608.05374}
}

• arXiv: 1608.05374