Wmt24 Indic Mt Eval

wmt24-indic-mt-eval

NLIP_Lab-IITH Low-Resource MT System for WMT24 Indic MT Shared Task — Sahoo et al. (2024) (arXiv:2410.03215, 2024)

What this evaluates

Evaluates machine translation quality for low-resource Northeast Indian languages (Assamese, Khasi, Mizo, Manipuri) paired with English. It probes cross-lingual transfer capabilities, model adaptation under data scarcity, and the effectiveness of architectural constraints like layer freezing and script-based language grouping.

Datasets

• IndicNECorp1.0 — total ?; splits: dev (-1), test (-1)

Metrics

• BLEU (primary) — range: percent
  • Standard n-gram overlap metric. Computed using SacreBLEU v2.3.1 with mixed case, no tokenization (13a), exponential smoothing, and nrefs:1.
• chrF — range: percent
  • Character n-gram F-score metric. Computed with a character n-gram order of 2.
• chrF++ — range: percent
  • Extension of chrF that incorporates word n-grams alongside character n-grams to improve correlation with human judgment.
• TER — range: percent
  • Translation Edit Rate measuring the minimum number of edits (insertions, deletions, substitutions, shifts) required to transform the hypothesis into the reference.
• RIBES — range: [0, 1]
  • Rank-based Index evaluating translation quality based on rank correlation of n-gram matches, robust to word order variations.
• COMET — range: [0, 1]
  • Cross-lingual Evaluation Framework using a pre-trained multilingual model to predict human-like quality scores based on source, reference, and hypothesis.

Input / output format

Input: Source sentence in English or a Northeast Indian language (Assamese, Khasi, Mizo, Manipuri).

Output: Translated target sentence in the corresponding target language.

Scoring recipe

import sacrebleu
# predictions and gold are lists of strings
bleu = sacrebleu.corpus_bleu(
    predictions, [gold],
    tokenize='none', smooth_method='exp',
    lowercase=False, force=False, lowercase_mixed=True,
    use_effective_order=True
)
return bleu.score

Common pitfalls

• SacreBLEU tokenization and smoothing settings must exactly match the paper's signature (tok:13a, smooth:exp, case:mixed) to reproduce scores.
• chrF word order parameter defaults to 6 in many implementations, but the paper explicitly sets it to 2.
• COMET and RIBES scores are highly sensitive to the specific model checkpoint and preprocessing pipeline used; default HuggingFace pipelines may diverge from reported values.

Evidence (verbatim from paper)

We evaluate using BLEU Papineni et al. ([2002]), chrF Popović ([2015]), and chrF++ Popović ([2017]) metrics. We use the SacreBLEU toolkit Post ([2018]) to perform our evaluation777SacreBLEU signature: nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.3.1 with a chrF word order of 2. Additionally, as per the evaluation metrics used by the organizers, we report results on TER Snover et al. ([2006]), RIBES Isozaki et al. ([2010]), and COMET Rei et al. ([2022]) for our primary and contrastive submissions.

Citation

@misc{sahoo2024nliplabiith,
  title={NLIP_Lab-IITH Low-Resource MT System for WMT24 Indic MT Shared Task},
  author={Sahoo et al. (2024)},
  year={2024},
  note={arXiv:2410.03215}
}

• arXiv: 2410.03215