Mt Data Filtering Eval

mt-data-filtering-eval

There's no Data Like Better Data: Using QE Metrics for MT Data Filtering — Jan-Thorsten Peter et al. (2023) (arXiv:2311.05350, 2023)

What this evaluates

This evaluation protocol assesses how effectively Quality Estimation (QE) metrics can filter low-quality or noisy sentence pairs from large parallel corpora. It measures whether retaining only the top 50% of high-scoring pairs improves downstream Neural Machine Translation (NMT) performance compared to using the full corpus or alternative filtering baselines like BICLEANER.

Datasets

• WMT & IWSLT Evaluation Campaigns — total ?; splits: train (-1), test (-1)

Metrics

• COMET22 (primary) — range: [0, 100]
  • A neural machine translation quality estimation metric that scores translations based on contextualized representations of source, hypothesis, and reference. It is trained on human judgments and outputs a continuous score typically scaled to a 0-100 range for reporting.

Input / output format

Input: Source sentence and reference translation (for metric computation); source sentence (for NMT inference).

Output: Generated target sentence from the NMT model.

Scoring recipe

def evaluate_nmt(test_pairs, nmt_model, comet22_model):
    scores = []
    for src, ref in test_pairs:
        hyp = nmt_model.translate(src)
        score = comet22_model.predict(src, hyp, ref)
        scores.append(score)
    return sum(scores) / len(scores)

Common pitfalls

• QE metrics are asymmetric; the authors use the same direction's scores for both translation directions to save compute, which may introduce bias.
• Test sets are filtered to remove sentences longer than 128 tokens, significantly reducing the size of WMT23 paragraph-level evaluation sets.
• BLEURT is excluded as the primary metric to prevent overfitting, since the proposed BLEURTQE filtering method is built on it.

Evidence (verbatim from paper)

In the discussion of the results we focus on the evaluation using COMET22. Traditional metrics like BLEU and CHRF are consistently outperformed by neural metrics in the WMT metrics shared task (Freitag et al., 2022a), thus we favor the use of such new metrics. We chose COMET22 over BLEURT in order to avoid overfitting on this last metric, as our proposed BLEURTQE model is based on it, and it also guides the checkpoint selection.

Citation

@misc{peter2023betterdata,
  title={There's no Data Like Better Data: Using QE Metrics for MT Data Filtering},
  author={Jan-Thorsten Peter et al. (2023)},
  year={2023},
  note={arXiv:2311.05350}
}

• arXiv: 2311.05350