Hintedbt Eval

hintedbt-eval

HintedBT: Augmenting Back-Translation with Quality and Transliteration Hints — Ramnath et al. (2021) (arXiv:2109.04443, 2021)

What this evaluates

Evaluates cross-script machine translation quality for low-resource Indian languages (Hindi, Gujarati, Tamil) translating to English. It specifically probes how well models leverage back-translation augmented with quality and transliteration hints to handle noisy data and script conversion challenges.

Datasets

• IIT Bombay en-hi Corpus — total 1680000; splits: train (1680000)
• WMT-2019 gu-en, TED2020, GNOME & Ubuntu, OPUS — total 162000; splits: train (162000)
• WMT-2020 ta-en, GNOME, OPUS — total 630000; splits: train (630000)
• WMT-2014 hi→en test — total 2500; splits: dev (520), test (2500)
• WMT-2019 gu→en test — total 1000; splits: dev (3400), test (1000)
• WMT-2020 ta→en test — total 1000; splits: dev (2000), test (1000)

Metrics

• SacreBLEU (primary) — range: percent
  • Standard n-gram precision-based metric computed using the SacreBLEU toolkit. It uses tokenized text (WordPiece) and applies sentence-level BLEU averaged over the test set.

Input / output format

Input: Source sentence in Hindi, Gujarati, or Tamil.

Output: Target sentence in English.

Scoring recipe

import sacrebleu
predictions = [pred.strip() for pred in model_outputs]
references = [[ref.strip()] for ref in gold_targets]
score = sacrebleu.corpus_bleu(predictions, references).score

Common pitfalls

• BLEU scores are computed on WordPiece tokenized text, not raw characters.
• Human evaluation uses a 0-6 scale for quality assessment but is not the primary reported metric.
• Back-translation data is generated from WMT Newscrawl 2013 English monolingual corpus, which is separate from the test sets.

Evidence (verbatim from paper)

We use SacreBLEU² (Post, 2018) to evaluate our models.

Citation

@misc{ramnath2021hintedbt,
  title={HintedBT: Augmenting Back-Translation with Quality and Transliteration Hints},
  author={Ramnath et al. (2021)},
  year={2021},
  note={arXiv:2109.04443}
}

• arXiv: 2109.04443