Nmt Low Resource Indonesian Eval

nmt-low-resource-indonesian-eval

Replicable Benchmarking of Neural Machine Translation (NMT) on Low-Resource Local Languages in Indonesia — Susanto et al. (2023) (arXiv:2311.00998, 2023)

What this evaluates

Evaluates neural machine translation performance across eight translation directions involving Indonesian and four low-resource Indonesian local languages (Javanese, Sundanese, Minangkabau, Balinese). It probes how different training paradigms (unsupervised, semi-supervised) and data augmentation strategies impact translation quality when parallel data is scarce.

Datasets

• Indonesian Local Language NMT Corpus — total ?; splits: test (-1)

Metrics

• spm200BLEU (primary) — range: other
  • Sentence-level BLEU score computed on text tokenized using a SentencePiece vocabulary of 200,000 subwords. It measures n-gram overlap between the model's translation and the reference, typically with a geometric mean of unigrams to 4-grams.

Input / output format

Input: Source sentence in Indonesian or a local Indonesian language (Javanese, Sundanese, Minangkabau, or Balinese).

Output: Translated target sentence in the corresponding local language or Indonesian.

Scoring recipe

def compute_spm200bleu(predictions, references):
    pred_tok = [spm200.tokenize(p) for p in predictions]
    ref_tok = [[spm200.tokenize(r)] for r in references]
    scores = [sentence_bleu(p, r) for p, r in zip(pred_tok, ref_tok)]
    return mean(scores) * 100

Common pitfalls

• The metric is specifically spm200BLEU (SentencePiece 200k vocabulary), not standard BLEU, making direct comparison with other NMT benchmarks difficult.
• Performance is highly asymmetric across translation directions (e.g., id->jv vs jv->id) and heavily dependent on the availability of parallel training data, so reporting a single aggregate score without specifying direction and language pair is misleading.
• Synthetic data augmentation (AUG) is only applied to semi-supervised/supervised training paradigms and does not affect unsupervised training results.

Evidence (verbatim from paper)

The results of these experiments (all metrics are inspm200BLEU, shown in Table 3), reveal a consistent trend: CodeXL approach results in a significantly better performing NMT systems compared to Scratch and PreXL.

Citation

@misc{susanto2023replicable,
  title={Replicable Benchmarking of Neural Machine Translation (NMT) on Low-Resource Local Languages in Indonesia},
  author={Susanto et al. (2023)},
  year={2023},
  note={arXiv:2311.00998}
}

• arXiv: 2311.00998