# Wmt Ape Eval

> Evaluates a model's ability to perform Automatic Post-Editing (APE) by correcting machine-translated German sentences using the original source text. It probes error detection, grammatical correction, and word-copying capabilities in a multi-source sequence-to-sequence setting. Use when the user wants to benchmark on WMT APE, or asks about evaluating this task. Reports case-sensitive BLEU.

- Skill: `qhjqhj00/wmt-ape-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/wmt-ape-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/wmt-ape-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/wmt-ape-eval

---


# wmt-ape-eval

> Learning to Copy for Automatic Post-Editing — Huang et al. (2019) (arXiv:1911.03627, 2019)

## What this evaluates

Evaluates a model's ability to perform Automatic Post-Editing (APE) by correcting machine-translated German sentences using the original source text. It probes error detection, grammatical correction, and word-copying capabilities in a multi-source sequence-to-sequence setting.

## Datasets

- **WMT APE** — total ?; splits: train (23000), dev2016 (1000), test2016 (2000), test2017 (2000), dev2018 (1000)

## Metrics

- `case-sensitive BLEU` **(primary)** — range: [0, 100]
  - Standard n-gram precision with brevity penalty, computed using multi-bleu.perl. Evaluated case-sensitively as per WMT APE official guidelines.
- `TER` — range: percent
  - Translation Edit Rate: minimum number of edits (insertions, deletions, substitutions, shifts) required to change the hypothesis into the reference, divided by reference length. Calculated using TERcom.

## Input / output format

**Input**: Pair of sequences: original source sentence (src) and machine-translated sentence (mt).

**Output**: Corrected German sentence (post-edit, pe).

## Scoring recipe

```python
bleu = multi_bleu.perl(predictions, references)  # case-sensitive
ter = TERcom(predictions, references)            # lower score is better
return {'bleu': bleu, 'ter': ter}
```

## Common pitfalls

- BLEU is computed case-sensitively, unlike standard lowercased BLEU, which significantly impacts scores.
- The dataset distinguishes between PBSMT and NMT variants with different training sizes; results must be compared within the same task variant.
- Ensemble models report higher scores than single models; comparisons must specify which variant is used.

## Evidence (verbatim from paper)

> We used the same evaluation metrics as the official WMT APE task (Chatterjee et al., 2018): casesensitive BLEU and TER. BLEU is computed by multi-bleu.perl (Koehn et al., 2007). TER is calculated using TERcom.

## Citation

```bibtex
@misc{huang2019learning,
  title={Learning to Copy for Automatic Post-Editing},
  author={Huang et al. (2019)},
  year={2019},
  note={arXiv:1911.03627}
}
```

- arXiv: 1911.03627