Indictrans2 Eval

indictrans2-eval

IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages — Gala et al. (2023) (arXiv:2305.16307, 2023)

What this evaluates

This benchmark evaluates multilingual machine translation quality across 22 scheduled Indian languages and English. It probes a model's ability to handle diverse domains (news, web, conversation, legal, etc.) and both Indic-to-English and English-to-Indic translation directions in an n-way parallel setting.

Datasets

• IN22 — total ?; splits: test (-1); repo https://github.com/AI4Bharat/IndicTrans2
• FLORES-200 — total ?; splits: test (-1); HF facebook/flores
• NTREX — total ?; splits: test (-1)
• WMT (2014, 2019, 2020) — total ?; splits: test (-1)
• WAT (2020, 2021) — total ?; splits: test (-1)
• UFAL — total ?; splits: test (-1)

Metrics

• chrF++ (primary) — range: [0, 100]
  • Character n-gram F-score extended with word unigrams and bigrams. Computed via sacreBLEU with character order 6, word order 2, and no space normalization.
• BLEU — range: [0, 100]
  • Standard n-gram precision metric. Computed using sacreBLEU with mteval-v13a tokenizer for Indic-to-English and IndicNLP/Urduhack tokenizers for English-to-Indic.
• COMET-DA — range: [0, 1]
  • Reference-based model-based metric using the COMET-22 DA model (XLM-RoBERTa backbone). Only reported for 13 Indic languages supported by the underlying XLM-R model.

Input / output format

Input: Source sentence in English or one of the 22 scheduled Indian languages.

Output: Translated target sentence in the corresponding target language.

Scoring recipe

def compute_chrF2_plus(predictions, references, direction):
    if direction == 'Indic-En':
        tok = 'mteval-v13a'
    else:
        tok = 'indicnlp_urduhack'
    
    tokenized_preds = tokenize(predictions, method=tok)
    tokenized_refs = tokenize(references, method=tok)
    
    score = sacrebleu.corpus_chrf(
        tokenized_preds, [tokenized_refs],
        char_order=6, word_order=2, space=False
    )
    return score.score

Common pitfalls

• Tokenization dependency: BLEU and chrF++ scores are highly sensitive to tokenization. Standard Moses tokenizers fail on Indic scripts; specific tokenizers (IndicNLP, Urduhack) must be used for En-Indic directions.
• COMET coverage gap: COMET-22 DA relies on XLM-RoBERTa and only supports 13 of the 22 Indic languages, leaving 9 languages unmeasured by this metric.
• Proprietary model opacity: Commercial systems (Google Translate, Azure, GPT-3.5) are evaluated via API without access to training data or internal parameters, making fair comparison an approximation.

Evidence (verbatim from paper)

In this work, we, therefore, primarily rely on chrF++ as our primary metric for evaluating translation quality. We also report additional metrics such as BLEU (Papineni et al., 2002) and COMET (Rei et al., 2022).

Citation

@misc{gala2023indictrans2,
  title={IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages},
  author={Gala et al. (2023)},
  year={2023},
  note={arXiv:2305.16307}
}

• arXiv: 2305.16307