Autshumato Nmt Eval

autshumato-nmt-eval

Benchmarking Neural Machine Translation for Southern African Languages — Martinus & Abbott (2019) (arXiv:1906.10511, 2019)

What this evaluates

Evaluates neural machine translation performance across five Southern African languages (Afrikaans, isiZulu, Northern Sotho, Setswana, Xitsonga) from English. It probes how dataset size and morphological complexity (e.g., agglutinative vs. non-agglutinative) impact translation quality in low-resource settings.

Datasets

• Autshumato — total ?; splits: test (-1); repo https://github.com/LauraMartinus/ukuxhumana

Metrics

• BLEU (primary) — range: percent
  • Standard n-gram overlap metric between machine-generated translations and human reference translations, computed with sentence-level averaging and a brevity penalty to penalize overly short outputs.

Input / output format

Input: English source sentences.

Output: Target language translation (Afrikaans, isiZulu, Northern Sotho, Setswana, or Xitsonga).

Scoring recipe

def compute_bleu(predictions, references):
    # predictions and references are lists of strings
    bleu_scores = []
    for pred, ref in zip(predictions, references):
        # Standard sentence-level BLEU with 4-gram precision and smoothing
        score = sentence_bleu([ref], pred, smoothing_method=SmoothingFunction().method1)
        bleu_scores.append(score)
    return sum(bleu_scores) / len(bleu_scores) * 100

Common pitfalls

• BLEU scores may not accurately reflect translation quality for highly agglutinative languages like isiZulu due to morphological richness and vocabulary mismatch.
• Dataset size and data quality are confounded; poor performance on isiZulu is attributed to both morphological complexity and low-quality parallel data, making it hard to isolate the cause.
• The paper reports baseline results on a newly released evaluation set without specifying exact train/val/test split sizes or tokenization details.

Evidence (verbatim from paper)

The BLEU scores for each model on English-to-Target language are presented in Table 2. The Transformer model outperformed the ConvS2S model for all languages. These results serve as initial baseline results for the given languages on the evaluation set.

Citation

@misc{martinus2019benchmarking,
  title={Benchmarking Neural Machine Translation for Southern African Languages},
  author={Martinus & Abbott (2019)},
  year={2019},
  note={arXiv:1906.10511}
}

• arXiv: 1906.10511