Imasc Eval

imasc-eval

IMaSC -- ICFOSS Malayalam Speech Corpus — Gopinath et al. (2022) (arXiv:2211.12796, 2022)

What this evaluates

Evaluates the perceptual quality and naturalness of synthesized Malayalam speech generated by a multi-speaker TTS model trained on the IMaSC corpus. It probes the model's ability to capture agglutinative morphology, phonemic orthography, and diverse prosodic styles through subjective human listening tests.

Datasets

• IMaSC — total 34473; splits: test (-1)

Metrics

• Mean Opinion Score (MOS) (primary) — range: [1, 5]
  • Average rating on a 5-point Likert scale for naturalness (5=excellent, 4=good, 3=fair, 2=poor, 1=bad) assigned by human listeners. Computed as the mean score across all participants and all evaluated audio samples.

Input / output format

Input: Malayalam text sentences (tokenized as characters) for synthesis; corresponding ground-truth audio samples for reference.

Output: Synthesized audio waveforms generated by the VITS TTS model.

Scoring recipe

def compute_mos(predictions, gold, num_participants=20):
    total_score = 0
    total_ratings = 0
    for sample in predictions:
        ratings = get_human_ratings(sample, num_participants)
        total_score += sum(ratings)
        total_ratings += num_participants
    return total_score / total_ratings

Common pitfalls

• The evaluation uses a very small test set (only 10 synthesized + 2 ground truth sentences per speaker), which may not capture full corpus diversity or speaker variability.
• MOS is highly subjective and sensitive to participant demographics, listening environment, and prior exposure to Malayalam, none of which are controlled or reported.
• The paper averages MOS across speakers and participants without reporting per-speaker variance or statistical significance testing.

Evidence (verbatim from paper)

We conducted a crowd-sourced Mean Opinion Score (MOS) test with 20 participants for evaluating the models. 10 sentences were randomly selected from the test dataset to be synthesized, and 2 sentences, along with their corresponding audio, were chosen for evaluating ground truth. In the survey, each speaker thus had 12 text-audio pairs to be evaluated, repeated for all 8 speakers for a total of 96 questions. The different audio samples were each scored on a 5-point scale for naturalness, with 5 being excellent, 4 being good, 3 being fair, 2 being poor, and 1 being bad.

Citation

@misc{gopinath2022imasc,
  title={IMaSC -- ICFOSS Malayalam Speech Corpus},
  author={Gopinath et al. (2022)},
  year={2022},
  note={arXiv:2211.12796}
}

• arXiv: 2211.12796