Phomt Eval

phomt-eval

PhoMT: A High-Quality and Large-Scale Benchmark Dataset for Vietnamese-English Machine Translation — Doan et al. (2021) (arXiv:2110.12199, 2021)

What this evaluates

This benchmark evaluates Vietnamese-English machine translation quality by comparing neural baselines and commercial engines. It probes translation accuracy across multiple domains and sentence lengths using both automatic metrics and human preference judgments.

Datasets

• PhoMT — total 3020000; splits: val (-1), test (-1); repo https://github.com/VinAIResearch/PhoMT

Metrics

• BLEU (primary) — range: [0, 100]
  • Case-sensitive BLEU score computed using SacreBLEU. Higher values indicate better translation quality.
• TER — range: [0, 100]
  • Translation Edit Rate (TER). Lower values indicate better performance.

Input / output format

Input: Source sentence in Vietnamese or English.

Output: Translated sentence in the target language (English or Vietnamese).

Scoring recipe

def compute_automatic_metrics(predictions, references):
    bleu = sacrebleu.corpus_bleu(predictions, [references])
    ter = tercom(predictions, references)  # or equivalent TER implementation
    return {'BLEU': bleu.score, 'TER': ter.score}

def compute_human_eval(predictions, references, annotators):
    # Sample 100 test sentences, shuffle 5 system outputs per sentence
    # 3 annotators pick best per sentence
    votes = count_votes(annotators.choices)
    return {sys: votes[sys]/100 for sys in systems}

Common pitfalls

• BLEU is computed case-sensitively using SacreBLEU, not standard tokenized BLEU.
• Human evaluation uses a 5-system shuffle protocol with 3 annotators per sentence, not a simple pairwise comparison.
• Automatic metrics (BLEU/TER) do not always correlate with human preference, especially for commercial engines vs. neural models.

Evidence (verbatim from paper)

We report standard metrics TER (Snover et al., 2006) and BLEU (Papineni et al., 2002), in which lower TER and higher BLEU indicate better performances. We compute the case-sensitive BLEU score using SacreBLEU (Post, 2018).

Citation

@misc{doan2021phomt,
  title={PhoMT: A High-Quality and Large-Scale Benchmark Dataset for Vietnamese-English Machine Translation},
  author={Doan et al. (2021)},
  year={2021},
  note={arXiv:2110.12199}
}

• arXiv: 2110.12199