Pmindia Nmt Eval

pmindia-nmt-eval

PMIndia -- A Collection of Parallel Corpora of Languages of India — Haddow et al. (2020) (arXiv:2001.09907, 2020)

What this evaluates

This evaluation probes the quality of automatic machine translation between English and 13 Indian languages using a parallel corpus. It measures how well NMT systems can handle diverse linguistic structures, including abugida scripts and agglutinative morphology, across low-resource language pairs.

Datasets

• PMIndia — total ?; splits: dev (1000), test (1000); repo https://github.com/bhoddow/pmindia-crawler

Metrics

• BLEU (primary) — range: percent
  • Standard n-gram precision-based metric computed over tokenized text. The paper explicitly uses the Moses multi-bleu.perl script rather than sacrebleu due to lack of support for South Asian scripts and languages.

Input / output format

Input: Source sentence in English or one of the 13 Indian languages, preprocessed with language-specific normalization, tokenization, and BPE subword splitting (10000 merges).

Output: Translated target sentence in the corresponding language, evaluated on the same tokenized/subword format.

Scoring recipe

def compute_bleu(predictions, references):
    # predictions, references: list of tokenized strings
    # Uses Moses multi-bleu.perl script
    import subprocess
    with open('pred.txt', 'w') as f:
        f.write('\n'.join(predictions))
    cmd = f"multi-bleu.perl {' '.join(references)} < pred.txt"
    out = subprocess.run(cmd, shell=True, capture_output=True, text=True)
    return float(out.stdout.split()[0])

Common pitfalls

• Using sacrebleu instead of the Moses multi-bleu.perl script, as sacrebleu lacks support for South Asian scripts and languages used in this benchmark.
• Relying solely on BLEU for agglutinative Dravidian languages, as word-based n-gram matching is known to be less reliable for morphologically rich languages.
• Evaluating on raw text without applying the specified BPE subword splits (10000 merges) and language-specific tokenization, which significantly affects score comparability.

Evidence (verbatim from paper)

The MT results (BLEU scores) are show in Table 5. We evaluated on tokenized text, using the multi-bleu.perl script from Moses, since sacrebleu (Post, 2018) does not currently support South Asian languages.

Citation

@misc{haddow2020pmindia,
  title={PMIndia -- A Collection of Parallel Corpora of Languages of India},
  author={Haddow et al. (2020)},
  year={2020},
  note={arXiv:2001.09907}
}

• arXiv: 2001.09907