Irish English St Eval

irish-english-st-eval

Leveraging Synthetic Audio Data for End-to-End Low-Resource Speech Translation — Moslem (2024) (arXiv:2406.17363, 2024)

What this evaluates

Evaluates end-to-end speech translation from Irish to English, specifically probing how synthetic audio data and augmentation techniques (noise, VAD) impact model performance in low-resource settings.

Datasets

• IWSLT-2023, FLEURS, Bitesize, SpokenWords — total ?; splits: val (-1), test (-1); repo https://github.com/ymoslem/MT-Preparation

Metrics

• chrF++ (primary) — range: [0, 1]
  • Character n-gram F-score computed over character-level n-grams (typically 1-6) between predicted and reference translations. Used here to select the best validation checkpoint during training.

Input / output format

Input: Raw Irish audio waveform

Output: English text transcript

Scoring recipe

def compute_chrf(predictions, references, n=6, beta=2.0):
    precisions = []
    recalls = []
    for i in range(1, n+1):
        p = count_ngram_overlap(predictions, references, i)
        r = count_ngram_overlap(references, predictions, i)
        precisions.append(p)
        recalls.append(r)
    p_mean = geometric_mean(precisions)
    r_mean = geometric_mean(recalls)
    return (1 + beta**2) * (p_mean * r_mean) / (beta**2 * p_mean + r_mean)

Common pitfalls

• Using a small early stopping patience on low-resource validation sets can prevent the model from seeing the full data distribution, hurting robustness.
• Synthetic audio generated via TTS lacks natural background noise and silence patterns, potentially inflating validation scores compared to real-world deployment.

Evidence (verbatim from paper)

We train the main models with Whisper Medium for at least two epochs, and save the best performing checkpoint based on the chrF++ score on the validation dataset.

Citation

@misc{moslem2024leveraging,
  title={Leveraging Synthetic Audio Data for End-to-End Low-Resource Speech Translation},
  author={Moslem (2024)},
  year={2024},
  note={arXiv:2406.17363}
}

• arXiv: 2406.17363