Wmt17 Nmt Benchmark Eval

wmt17-nmt-benchmark-eval

Sockeye 3: Fast Neural Machine Translation with PyTorch — Hieber et al. (2022) (arXiv:2207.05851, 2022)

What this evaluates

Evaluates the training efficiency, inference speed, and translation quality of neural machine translation systems on standard WMT17 benchmarks. It probes the trade-offs between model architecture, hardware acceleration (FP16/INT8), and batching strategies.

Datasets

• WMT17 English-German — total 5900000; splits: train (-1)
• WMT17 Russian-English — total 25000000; splits: train (-1)

Metrics

• BLEU (primary) — range: [0, 100]
  • Standard n-gram overlap metric for machine translation quality. Computed on the full test set after averaging the 8 best model checkpoints.
• Training Time (Hours) — range: hours
  • Wall-clock time required to complete the specified number of training updates (25K for En-De, 70K for Ru-En) on 8 GPUs.
• Translation Speed (Sent/Sec) — range: sent/sec
  • Number of source sentences processed per second during inference, averaged over 3 runs. Measured on GPU (g4dn.xlarge) and CPU (c5.2xlarge) with batch sizes 1 and 64.

Input / output format

Input: Source language sentences pre-processed with byte-pair encoding (32K operations) and filtered to a maximum length of 95 tokens.

Output: Target language sentences generated via greedy search or beam search.

Scoring recipe

train_time = measure_wall_clock_time(model, dataset, updates=25000, devices=8)
speed = measure_throughput(model, dataset, batch_size=64, device='gpu')
predictions = []
for ckpt in best_8_checkpoints:
    predictions.append(translate(dataset, ckpt))
final_predictions = average_predictions(predictions)
bleu = compute_bleu(gold, final_predictions)

Common pitfalls

• BLEU scores are computed after averaging the 8 best model checkpoints, not from a single checkpoint.
• Inference speed varies drastically with batch size (1 vs 64) and hardware (GPU vs CPU); results are not directly comparable across these settings.
• The exact BLEU implementation (e.g., Moses vs SacreBLEU) is not specified, which can cause minor score discrepancies.

Evidence (verbatim from paper)

We select two translation tasks for which pre-processed data sets are available: WMT17 English-German (5.9M sentences) and Russian-English (25M sentences). We further process the data by applying byte-pair encoding with 32K operations and filtering out sentences longer than 95 tokens. Models are trained for either 25K updates (En-De) or 70K updates (Ru-En) with checkpoints every 500 updates. The 8 best checkpoints are averaged to produce the final model weights. Shown in Table 4, Sockeye and Fairseq are fastest, training models with comparable BLEU scores in comparable time.

Citation

@misc{hieber2022sockeye3,
  title={Sockeye 3: Fast Neural Machine Translation with PyTorch},
  author={Hieber et al. (2022)},
  year={2022},
  note={arXiv:2207.05851}
}

• arXiv: 2207.05851