# Canmt Contrastive Eval

> Evaluates context-aware neural machine translation models on their ability to correctly translate context-dependent discourse phenomena (e.g., anaphoric pronouns, deixis, ellipsis) across sentence boundaries in concatenated input windows. Use when the user wants to benchmark on En→Ru movie subtitles (Voita et al., 2019), En→De TED talk subtitles (IWSLT17), Voita contrastive set, ContraPro, or asks about evaluating this task. Reports Contrastive accuracy.

- Skill: `qhjqhj00/canmt-contrastive-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/canmt-contrastive-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/canmt-contrastive-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/canmt-contrastive-eval

---


# canmt-contrastive-eval

> Encoding Sentence Position in Context-Aware Neural Machine Translation with Concatenation — Lupo et al. (2023) (arXiv:2302.06459, 2023)

## What this evaluates

Evaluates context-aware neural machine translation models on their ability to correctly translate context-dependent discourse phenomena (e.g., anaphoric pronouns, deixis, ellipsis) across sentence boundaries in concatenated input windows.

## Datasets

- **En→Ru movie subtitles (Voita et al., 2019)** — total ?; splits: test (-1); repo https://github.com/lorelupo/focused-concat
- **En→De TED talk subtitles (IWSLT17)** — total ?; splits: test (-1); repo https://github.com/lorelupo/focused-concat
- **Voita contrastive set** — total ?; splits: test (-1); repo https://github.com/lorelupo/focused-concat
- **ContraPro** — total ?; splits: test (-1); repo https://github.com/lorelupo/focused-concat

## Metrics

- `BLEU` — range: percent
  - Standard BLEU score computed via Moses' multi-bleu script. Detokenized for German, lowercased for Russian.
- `Contrastive accuracy` **(primary)** — range: percent
  - Percentage of correctly translated examples in the contrastive sets (Voita's set for En→Ru; ContraPro for En→De).

## Input / output format

**Input**: Sliding windows of 4 concatenated source sentences.

**Output**: Target language translation of the entire 4-sentence window.

## Scoring recipe

```python
# Compute BLEU
if lang == 'de':
    preds = detokenize(predictions)
else:
    preds = lower(predictions)
bleu_score = multi_bleu(preds, references)

# Compute Contrastive Accuracy
correct = sum(1 for pred, gold in zip(predictions, gold_contrastive) if pred == gold)
contrastive_acc = (correct / len(gold_contrastive)) * 100
```

## Common pitfalls

- BLEU is repeatedly shown to be ill-equipped to detect improvements in context-aware MT.
- Position encoding benefits on En→De fail to transfer due to an order-of-magnitude smaller training corpus (0.2M vs 6M sentences).
- Non-persistent position embeddings were dropped after preliminary experiments showed no benefit.

## Evidence (verbatim from paper)

> In addition to evaluating the average translation quality with BLEU... we employ two contrastive sets to evaluate the translation of context-dependent anaphoric pronouns. For En→Ru, we adopt Voita et al. (2019)’s set for the evaluation of inter-sentential deixis, lexical cohesion, verb-phrase ellipsis, and inflectional ellipsis. For En→De, we evaluate the models on the translation of context-dependent ambiguous pronouns with ContraPro (Müller et al., 2018)...

## Citation

```bibtex
@misc{lupo2023encoding,
  title={Encoding Sentence Position in Context-Aware Neural Machine Translation with Concatenation},
  author={Lupo et al. (2023)},
  year={2023},
  note={arXiv:2302.06459}
}
```

- arXiv: 2302.06459