Wmt19 Online Mt Selection Eval

wmt19-online-mt-selection-eval

Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort — Mendonça et al. (2021) (arXiv:2105.13385, 2021)

What this evaluates

Evaluates an online learning framework's ability to dynamically identify the highest-quality machine translation systems from an ensemble using minimal human feedback, and measures the sample efficiency (number of human assessments needed) to converge to the official top-performing systems.

Datasets

• WMT'19 News Translation — total 7711; splits: test (7711)

Metrics

• convergence_to_top3 (primary) — range: iterations
  • The number of test segments processed (iterations) until the online algorithm's weighted ranking of MT systems matches the official WMT'19 top-3 systems for that language pair. Lower values indicate faster convergence with less human effort.

Input / output format

Input: A source sentence, a reference translation, and automatic translations from N competing MT systems for a specific language pair, along with available human assessment scores or fallback metric scores.

Output: A probability/weight distribution over the N competing MT systems, updated sequentially after each segment's feedback is received.

Scoring recipe

def compute_convergence(predictions_weights, gold_top3, max_iters=7711):
    for t, weights in enumerate(predictions_weights):
        top_n = sorted(weights, key=weights.get, reverse=True)[:3]
        if set(top_n) == gold_top3:
            return t + 1
    return max_iters

Common pitfalls

• Fallback strategies (human-zero, human-avg, human-comet) are used when human scores are missing, drastically altering the loss signal and convergence speed.
• The test set is shuffled once per run, and EXP3 requires averaging over 10 runs due to stochastic arm selection, making results sensitive to random seeds and shuffling.

Evidence (verbatim from paper)

The main questions addressed by our experiment are: (i) whether an online learning approach can give a greater weight to the top performing systems for each language pair according to the shared task's official ranking, and (ii) if so, how quickly (i.e., how many translations need to be assessed by human evaluators in order to find the best system).

Citation

@misc{mendonca2021onlinelearningmt,
  title={Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effort},
  author={Mendonça et al. (2021)},
  year={2021},
  note={arXiv:2105.13385}
}

• arXiv: 2105.13385