Canmt Contrastive Eval

canmt-contrastive-eval

Encoding Sentence Position in Context-Aware Neural Machine Translation with Concatenation — Lupo et al. (2023) (arXiv:2302.06459, 2023)

What this evaluates

Evaluates context-aware neural machine translation models on their ability to correctly translate context-dependent discourse phenomena (e.g., anaphoric pronouns, deixis, ellipsis) across sentence boundaries in concatenated input windows.

Datasets

• En→Ru movie subtitles (Voita et al., 2019) — total ?; splits: test (-1); repo https://github.com/lorelupo/focused-concat
• En→De TED talk subtitles (IWSLT17) — total ?; splits: test (-1); repo https://github.com/lorelupo/focused-concat
• Voita contrastive set — total ?; splits: test (-1); repo https://github.com/lorelupo/focused-concat
• ContraPro — total ?; splits: test (-1); repo https://github.com/lorelupo/focused-concat

Metrics

• BLEU — range: percent
  • Standard BLEU score computed via Moses' multi-bleu script. Detokenized for German, lowercased for Russian.
• Contrastive accuracy (primary) — range: percent
  • Percentage of correctly translated examples in the contrastive sets (Voita's set for En→Ru; ContraPro for En→De).

Input / output format

Input: Sliding windows of 4 concatenated source sentences.

Output: Target language translation of the entire 4-sentence window.

Scoring recipe

# Compute BLEU
if lang == 'de':
    preds = detokenize(predictions)
else:
    preds = lower(predictions)
bleu_score = multi_bleu(preds, references)

# Compute Contrastive Accuracy
correct = sum(1 for pred, gold in zip(predictions, gold_contrastive) if pred == gold)
contrastive_acc = (correct / len(gold_contrastive)) * 100

Common pitfalls

• BLEU is repeatedly shown to be ill-equipped to detect improvements in context-aware MT.
• Position encoding benefits on En→De fail to transfer due to an order-of-magnitude smaller training corpus (0.2M vs 6M sentences).
• Non-persistent position embeddings were dropped after preliminary experiments showed no benefit.

Evidence (verbatim from paper)

In addition to evaluating the average translation quality with BLEU... we employ two contrastive sets to evaluate the translation of context-dependent anaphoric pronouns. For En→Ru, we adopt Voita et al. (2019)’s set for the evaluation of inter-sentential deixis, lexical cohesion, verb-phrase ellipsis, and inflectional ellipsis. For En→De, we evaluate the models on the translation of context-dependent ambiguous pronouns with ContraPro (Müller et al., 2018)...

Citation

@misc{lupo2023encoding,
  title={Encoding Sentence Position in Context-Aware Neural Machine Translation with Concatenation},
  author={Lupo et al. (2023)},
  year={2023},
  note={arXiv:2302.06459}
}

• arXiv: 2302.06459