Ethiomt Eval

ethiomt-eval

EthioMT: Parallel Corpus for Low-resource Ethiopian Languages — Tonja et al. (2024) (arXiv:2403.19365, 2024)

What this evaluates

Machine translation performance across multiple low-resource Ethiopian languages paired with English, evaluating both English-to-Ethiopian and Ethiopian-to-English directions. It probes how model initialization (training from scratch vs. fine-tuning a multilingual model) and available corpus size impact translation quality.

Datasets

• EthioMT — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• spBLEU (primary) — range: [0, 100]
  • Sentence-level BLEU score computed using the SacreBLEU library, which standardizes tokenization and reference handling. It is typically reported on a 0-100 scale.

Input / output format

Input: Source sentence in an Ethiopian language or English, paired with a target sentence in English or an Ethiopian language respectively.

Output: Translated sentence in the target language.

Scoring recipe

import sacrebleu
# predictions: list of generated translations
# references: list of lists of reference translations
score = sacrebleu.corpus_bleu(predictions, [references]).score

Common pitfalls

• Performance varies drastically with corpus size, so low scores for some languages may reflect data scarcity rather than model failure.
• Fine-tuning a pre-trained multilingual model (m2m100) yields significantly higher scores than training a transformer from scratch, making baseline selection critical for fair comparison.
• Absolute spBLEU scores are extremely low (0.01–17.8), indicating that even the best models struggle with these low-resource languages.

Evidence (verbatim from paper)

We evaluated the above approaches in bidirectional translation from Ethiopian languages to English and From English to Ethiopian languages. We used Sacrebleu Post ([2018]) evaluation metrics to evaluate translation models. ... the performance of the transformer model also varies in the ranges of 0.01 – 17.8 spBLEU from language to language with different corpus sizes.

Citation

@misc{tonja2024ethiomt,
  title={EthioMT: Parallel Corpus for Low-resource Ethiopian Languages},
  author={Tonja et al. (2024)},
  year={2024},
  note={arXiv:2403.19365}
}

• arXiv: 2403.19365