Hinmix Robust Cmt Eval

hinmix-robust-cmt-eval

Synthetic Data Generation and Joint Learning for Robust Code-Mixed Translation — Kartik et al. (2024) (arXiv:2403.16771, 2024)

What this evaluates

Evaluates machine translation models on code-mixed and noisy Hindi-English and Bengali-English text, measuring robustness to script variations, romanization, and synthetic noise. The protocol tests both in-domain performance on the HINMIX corpus and out-of-domain generalizability on LinCE, SpokenTutorial, and IITB Hi-En. It also assesses zero-shot transfer to unseen code-mixed Bengali-English translation.

Datasets

• HINMIX — total ?; splits: test (-1); repo https://github.com/LCS2-IIITD/Robust_CodeMIX_MT

Metrics

• SacreBLEU (primary) — range: [0, 100]
  • Standard BLEU score computed with the SacreBLEU toolkit, using standard tokenization, corpus-level aggregation, and default smoothing.
• METEOR — range: [0, 100]
  • Alignment-based metric that computes precision, recall, and penalty for fragmentation, incorporating synonymy matching and stemming.

Input / output format

Input: Source sentence in code-mixed Hindi/English or Bengali/English, presented in Devanagari, Romanized, or noisy (typo/phonetic) variants.

Output: Target English sentence.

Scoring recipe

# SacreBLEU
bleu = sacrebleu.corpus_bleu(predictions, [references])
# METEOR
meteor = meteor_score(predictions, references)
# Report both as percentages (0-100)

Common pitfalls

• Baselines are fine-tuned from scratch on HINMIX for fair comparison, not evaluated in a zero-shot or pre-trained setting.
• Noise robustness is tested on three distinct corpus variants (clean code-mixed, romanized, and noisy), not a single mixed test set.
• Zero-shot Bengali evaluation uses a completely unseen language pair without any parallel code-mixed Bengali training data.

Evidence (verbatim from paper)

Finally, we evaluate the quality of models on SacreBLEU (Ott et al., 2019) and METEOR (Banerjee and Lavie, 2005) metrics.

Citation

@misc{kartik2024synthetic,
  title={Synthetic Data Generation and Joint Learning for Robust Code-Mixed Translation},
  author={Kartik et al. (2024)},
  year={2024},
  note={arXiv:2403.16771}
}

• arXiv: 2403.16771