Multimodal Mt Rl Eval

multimodal-mt-rl-eval

Multimodal Machine Translation with Reinforcement Learning — Qian et al. (2018) (arXiv:1805.02356, 2018)

What this evaluates

Evaluates a multimodal sequence-to-sequence model's ability to generate accurate translations conditioned on both source text and image features. It specifically probes whether reinforcement learning with BLEU-based rewards can mitigate exposure bias and improve translation quality over standard supervised maximum likelihood estimation.

Datasets

• WMT17 multimodal machine translation shared task — total ?; splits: train (29000), val (1000), test (1071)

Metrics

• BLEU (primary) — range: [0, 100]
  • Computes n-gram precision up to order 4 between generated and reference translations, multiplied by a brevity penalty to penalize short outputs. Corpus-level BLEU is reported as the final metric.
• Perplexity — range: [0, ∞)
  • Measures how well the model predicts the reference sequence, defined as exp(mean(-log p(x))). Lower values indicate better predictive performance.

Input / output format

Input: English source sentence concatenated with 2048-dimensional ResNet-50 image features.

Output: Target language sentence (German, French, or Czech) generated via greedy search.

Scoring recipe

def compute_corpus_bleu(preds, refs):
    precisions = [count_ngram_matches(preds, refs, n) for n in range(1, 5)]
    brevity_penalty = min(1.0, len(preds) / len(refs))
    return brevity_penalty * (prod(precisions) ** 0.25) * 100

def compute_perplexity(model, data):
    log_probs = [model.log_prob(x) for x in data]
    return exp(mean(log_probs))

Common pitfalls

• The RL training loop optimizes using sentence-level BLEU rewards, but the final reported performance metric is corpus-level BLEU, which can yield different model rankings.
• Image features are fixed ResNet-50 outputs rather than end-to-end learned representations, potentially introducing noise from irrelevant objects in the images.
• Perplexity is reported alongside BLEU but is not the primary optimization target; models with lower perplexity do not always achieve higher BLEU scores.

Evidence (verbatim from paper)

We use BLEU score [9] (ngram of up to 4) as our major automatic evaluation metrics. BLEU score is the n-gram overlap between output translation and reference translation. And a brevity term is added to penalize short translations.

Citation

@misc{qian2018multimodal,
  title={Multimodal Machine Translation with Reinforcement Learning},
  author={Qian et al. (2018)},
  year={2018},
  note={arXiv:1805.02356}
}

• arXiv: 1805.02356