Scb Mt En Th 2020 Eval

scb-mt-en-th-2020-eval

scb-mt-en-th-2020: A Large English-Thai Parallel Corpus — Lalita Lowphansirikul et al. (arXiv:2007.03541, 2020)

What this evaluates

This protocol evaluates neural machine translation quality between English and Thai. It measures translation accuracy on a newly curated 1M-parallel corpus (SCB_1M) and a filtered OPUS corpus, while also testing cross-domain generalization on the IWSLT 2015 Thai-English benchmark.

Datasets

• SCB_1M — total 1001752; splits: train (-1), val (-1), test (100177)
• MT_OPUS — total 3318153; splits: train (-1), val (-1), test (297874)
• IWSLT 2015 Thai-English — total 4242; splits: test (4242)

Metrics

• SacreBLEU (primary) — range: percent
  • Corpus-level BLEU score computed with the SacreBLEU toolkit. Uses 13a tokenization, exponential smoothing, and exactly one reference per hypothesis. Case handling is mixed-case for Thai-to-English and lowercase for English-to-Thai.

Input / output format

Input: Source sentence in either Thai or English.

Output: Target sentence in either English or Thai.

Scoring recipe

import sacrebleu

def compute_bleu(hypotheses, references, direction):
    if direction == 'th_en':
        return sacrebleu.corpus_bleu(
            hypotheses, references,
            lowercase=False, tokenize='13a',
            smooth_method='exp', smooth_value=0
        ).score
    else: # en_th
        return sacrebleu.corpus_bleu(
            hypotheses, references,
            lowercase=True, tokenize='13a',
            smooth_method='exp', smooth_value=0
        ).score

Common pitfalls

• Detokenization must strictly follow direction-specific rules: Moses detokenizer for word-level th→en, SentencePiece for subword outputs, and space-joining for word-level en→th.
• Cross-domain evaluation (training on SCB_1M, testing on MT_OPUS or vice versa) yields drastically lower BLEU scores (6–13) compared to in-domain evaluation (25–42), which can mislead model selection if splits are not stratified by source domain.
• Case sensitivity conventions differ between directions in the reported results, affecting direct score comparisons across language pairs.

Evidence (verbatim from paper)

SacreBLEU [Post, 2018] is used to evaluate translation quality in both directions. For th → en translation, word-level outputs are detokenized with Moses detokenizer and subword outputs for both Thai and English are detokenized Sentencepiece [Kudo and Richardson, 2018]. The version string used for computing BLEU score for case-sensitive and case-insertive are BLEU + case.mixed + numrefs.1 + smooth.exp + tok.13a + version.1.2.10 and BLEU + case.lc + numrefs.1 + smooth.exp + tok.13a + version.1.2.12 respectively.

Citation

@misc{lowphansirikul2020scbmtenth2020,
  title={scb-mt-en-th-2020: A Large English-Thai Parallel Corpus},
  author={Lalita Lowphansirikul et al.},
  year={2020},
  note={arXiv:2007.03541}
}

• arXiv: 2007.03541