Instruction Nmt Eval

instruction-nmt-eval

On Instruction-Finetuning Neural Machine Translation Models — Raunak et al. (2024) (arXiv:2410.05553, 2024)

What this evaluates

Evaluates whether neural machine translation models can follow diverse natural language instructions (e.g., formality, voice, casing, simplification) without task-specific retraining, while maintaining general translation quality.

Datasets

• WMT'20 News Translation (EN-DE) — total ?; splits: test (-1)
• Multi-30K — total 29000; splits: train (29000)
• Custom Instruction Dataset — total 30000; splits: finetune (27000), eval (3000)

Metrics

• RR (%) (primary) — range: percent
  • Resolution Rate (RR) measures the percentage of test instances where the model's output successfully satisfies the given natural language instruction constraint.
• ChrF_instruction — range: [0, 1]
  • Character n-gram F-score computed on the instruction-modified translation output to evaluate stylistic and constraint adherence.
• BLEU — range: [0, 1]
  • Standard n-gram precision-based metric for general translation quality, computed via SacreBLEU.
• ChrF2 — range: [0, 1]
  • Character n-gram F-score (order 2) for general translation quality.
• TER — range: [0, 1]
  • Translation Edit Rate measuring the number of edits required to transform the hypothesis into the reference.
• COMET — range: [0, 1]
  • Reference-based neural metric that scores translation quality using a pretrained model.
• Formal Accuracy — range: percent
  • Percentage of translations correctly classified as formal by an external evaluator or classifier.
• Informal Accuracy — range: percent
  • Percentage of translations correctly classified as informal by an external evaluator or classifier.

Input / output format

Input: Source sentence concatenated with a natural language instruction (or image tokens for multimodal tasks) prepended to the source text.

Output: Translated target sentence that adheres to the specified instruction constraints.

Scoring recipe

def compute_metrics(predictions, instructions, references):
    rr_correct = sum(1 for p, i in zip(predictions, instructions) if satisfies_constraint(p, i))
    rr = (rr_correct / len(predictions)) * 100
    chrf_inst = sacrechrf(predictions, references).score
    return {'RR (%)': rr, 'ChrF_instruction': chrf_inst}

Common pitfalls

• The instruction evaluation uses a separate 10% held-out split of the custom 30-task dataset, not the WMT'20 test set.
• Zero-shot composition is evaluated on instruction combinations never seen during the 90% finetuning phase.
• General translation metrics (BLEU/ChrF2) are measured on WMT'20, while instruction-specific metrics (RR/ChrF_instruction) are measured on the task-specific 1K-sample test sets.

Evidence (verbatim from paper)

The metrics BLEU, ChrF2, TER Papineni et al. ([2002]); Popović ([2015]); Snover et al. ([2006]) for the trained model on the WMT’20 validation and test sets (under beam size of $1$) as measured using SacreBLEU Post ([2018]) are presented in Appendix B, alongside reference-based COMET Rei et al. ([2020]) scores.

Citation

@misc{raunak2024instructionfinetuning,
  title={On Instruction-Finetuning Neural Machine Translation Models},
  author={Raunak et al. (2024)},
  year={2024},
  note={arXiv:2410.05553}
}

• arXiv: 2410.05553