Indic Nmt Eval

indic-nmt-eval

Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages — Gowtham Ramesh et al. (2021) (arXiv:2104.05596, 2021)

What this evaluates

Evaluates neural machine translation models for Indic languages by measuring translation quality against reference texts across multiple standard benchmarks. It specifically tests the effectiveness of training on the Samanantar parallel corpus compared to commercial systems and existing open-source baselines.

Datasets

• WAT2020 Indic task — total ?; splits: test (-1)
• WAT2021 Multi-IndicMT task — total ?; splits: test (-1)
• WMT test sets (2014, 2019, 2020) — total ?; splits: test (-1)
• UFAL Entam — total ?; splits: test (-1)
• FLORES test set — total ?; splits: test (-1)
• PMIndia en-as testset — total 3000; splits: val (1000), test (2000)

Metrics

• BLEU (SacreBLEU) (primary) — range: percent
  • SacreBLEU corpus score computed with direction-specific tokenization: mteval-v13a tokenizer for Indic→English, and IndicNLP tokenizer followed by SacreBLEU's default tokenizer for English→Indic.

Input / output format

Input: Source sentence in English or an Indic language

Output: Translated target sentence

Scoring recipe

def compute_bleu(preds, refs, direction):
    if direction == 'indic_to_en':
        tok = mteval_v13a_tokenize
    else:
        tok = lambda x: sacrebleu_tokenize(indicnlp_tokenize(x))
    return sacrebleu.corpus_bleu(
        [tok(p) for p in preds],
        [[tok(r) for r in ref] for ref in refs]
    ).score

Common pitfalls

• Tokenization differs by direction: Indic→English uses mteval-v13a, while English→Indic requires IndicNLP tokenizer before SacreBLEU.
• Strict train/test separation must be enforced when training baselines on existing parallel data to prevent data leakage.

Evidence (verbatim from paper)

We use BLEU scores for evaluating the models. To ensure consistency and reproducibility across the models, we provide SacreBLEU signatures in the footnote for Indic-English and English-Indic evaluations. For Indic-English, we use the in-built, default mteval-v13a tokenizer. For En-Indic, since SacreBLEU tokenizer does not support Indic languages, we first tokenize using the IndicNLP tokenizer before running SacreBLEU.

Citation

@misc{ramesh2021samanantar,
  title={Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages},
  author={Gowtham Ramesh et al. (2021)},
  year={2021},
  note={arXiv:2104.05596}
}

• arXiv: 2104.05596